Image generating apparatus generating reconstructed image, method, and computer-readable recording medium

ABSTRACT

A reconstructed image which further reflects the photo shooting information of a main object is generated. A layer determiner defines a layer of a reconstructed image. A layer image generator reconstructs an image of an object included in the allocated depth range from a light field image and the depth map of the light field image for each layer. A conversion pixel extractor extracts corresponding pixels on a conversion layer which corresponds to an object to be modified. The object to be modified is designated by an operation acquired by a modification operation acquirer. A reconstructed image generator converts layer images using a conversion matrix defined by a conversion matrix determiner and generates an image whose composition has been modified by superimposing the layer images.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Patent Application No.2012-039405, filed on Feb. 24, 2012, and the Japanese Patent ApplicationNo. 2012-062568, filed on Mar. 19, 2012, the entire disclosures of whichare incorporated by reference herein.

FIELD

This application relates to an image reconstructing technique.

BACKGROUND

After taking a photograph, there are cases where users would like tochange a composition of an object. In order to meet such a demand, thetechnique which changes an image after the image generation is known inthe field of image processing.

In relation to this technique, National Patent Publication No.2005-509985 discloses a method which defines an image area and modifiesthe composition of the image using interpolation in order to suppress asense of discomfort. The technique described in the National PatentPublication No. 2005-509985 rectifies the gap generated by thecomposition modification using the pixel values of the surroundingpixels.

Moreover, a technique, which acquires information regarding theposition, direction, and intensity of the light beams coming into thelens of a camera from an object by obtaining an image of an object shotfrom different viewpoints, is known. In relation to such a technique,the National Patent Publication No. 2008-515110 discloses a techniquewhich acquires a plurality of images (light field images) in which anobject is shot, and, from the plurality of images, the techniquereconstructs the images of the object whose focal distances, depths ofthe field and the like are varied.

The information regarding the object included in the images shot fromone viewpoint is limited to light beam arriving positions and lightintensity at those positions (corresponding to the coordinates of eachpixel and their pixel values, respectively). Thus, if an objectcomposition is modified on a planar image, even if the object ismodified using the technique described in the National PatentPublication No. 2005-509985, an interpolation not using the light beaminformation from an object is needed.

On the other hand, the National Patent Publication No. 2008-515110describes a technique of acquiring information which shows athree-dimensional form of an object by photo shooting the object from aplurality of viewpoints, and a technique of acquiring light beaminformation and the like, from an object hidden and invisible from acertain viewpoint. However, the National Patent Publication No.2008-515110 does not mention a method of generating the reconstructedimage in which the composition of an object has been changed, and doesnot solve the above-mentioned problem.

SUMMARY

The present invention is made in view of the above-mentioned situation,and an object of the present invention is to provide an image generatingapparatus generating a reconstructed image which further reflects thephoto shooting information regarding a main object, a method and acomputer-readable recording medium.

An image generating apparatus of a first aspect of the present inventioncomprises: an image acquirer which acquires a photo shooting imageformed by a plurality of sub images shot from each of a plurality ofviewpoints; an extractor which extracts, from sub-pixels forming the subimages, sub-pixels corresponding to an object whose composition is to bemodified as pixels to be modified; a modifier which modifies acorrespondence relationship with sub-pixels extracted as the pixels tobe modified among correspondence relationships between the sub-pixelsand pixels to be reconstructed forming a reconstructed image which isdefined on a predetermined reconstruction plane; and a generator whichgenerates a reconstructed image in which the composition of the objecthas been modified by calculating pixel values of the pixels to bereconstructed from pixel values of the sub-pixels using the modifiedcorrespondence relationship.

An image generating apparatus of a second aspect of the presentinvention comprises: an image acquirer which acquires a photo shootingimage formed by a plurality of sub images shot from each of a pluralityof viewpoints; an extractor which extracts, from sub-pixels forming thesub images, sub-pixels corresponding to an object to be deleted aspixels to be deleted; a modifier which modifies a correspondencerelationship between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane so that a degree of correspondence with thesub-pixels extracted as the pixels to be deleted is reduced; and agenerator which generates a reconstructed image having a small effect ofthe pixels to be deleted by calculating pixel values of the pixels to bereconstructed from pixel values of the sub-pixels using the modifiedcorrespondence relationship.

A method of a third aspect of the present invention comprises the stepsof: acquiring a photo shooting image formed by a plurality of sub imagesshot from each of a plurality of viewpoints; extracting, from sub-pixelsforming the sub images, sub-pixels corresponding to an object whosecomposition is to be modified as pixels to be modified; modifying acorrespondence relationship with sub-pixels extracted as the pixels tobe modified among correspondence relationships between the sub-pixelsand pixels to be reconstructed forming a reconstructed image which isdefined on a predetermined reconstruction plane; and generating areconstructed image in which the composition of the object has beenmodified by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship.

A method of a fourth aspect of the present invention comprising thesteps of: acquiring a photo shooting image formed by a plurality of subimages shot from each of a plurality of viewpoints; extracting, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectto be deleted as pixels to be deleted; modifying a correspondencerelationship between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane so that a degree of correspondence with thesub-pixels extracted as the pixels to be deleted is reduced; andgenerating a reconstructed image having a small effect of the pixels tobe deleted by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship.

A non-transitory computer-readable recording medium of a fifth aspect ofthe present invention has stored thereon a program executable by acomputer, the program controlling the computer to perform functionscomprising: acquiring a photo shooting image formed by a plurality ofsub images shot from each of a plurality of viewpoints; extracting, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectwhose composition is to be modified as pixels to be modified; modifyinga correspondence relationship with sub-pixels extracted as the pixels tobe modified among correspondence relationships between the sub-pixelsand pixels to be reconstructed forming a reconstructed image which isdefined on a predetermined reconstruction plane; and generating areconstructed image in which the composition of the object has beenmodified by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship.

A non-transitory computer-readable recording medium of a sixth aspect ofthe present invention has stored thereon a program executable by acomputer, the program controlling the computer to perform functionscomprising: acquiring a photo shooting image formed by a plurality ofsub images shot from each of a plurality of viewpoints; extracting, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectto be deleted as pixels to be deleted; modifying a correspondencerelationship between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane so that a degree of correspondence with thesub-pixels extracted as the pixels to be deleted is reduced; andgenerating a reconstructed image having a small effect of the pixels tobe deleted by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained whenthe following detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 is a diagram showing a configuration of a digital camera;

FIG. 2 is an illustration showing a configuration of an optical systemof a digital camera;

FIG. 3A shows a conceptual diagram of a light field image;

FIG. 3B shows an example of a light field image;

FIG. 3C shows an example of a light field depth map;

FIG. 4 is an illustration for explaining a light beam trace;

FIG. 5 is a list showing an example of a corresponding list;

FIG. 6A is a diagram showing a physical configuration of an imagegenerating apparatus of an embodiment 1;

FIG. 6B is a diagram showing a functional configuration of imagegenerating apparatus;

FIG. 7A is an illustration showing objects regarding the embodiment 1;

FIG. 7B is an illustration showing layers and a reconstructed image;

FIG. 7C is an illustration showing a reconstructed depth map;

FIG. 8A to FIG. 8G are illustrations for explaining operations ofinstructing composition modifications regarding the embodiment 1, FIG.8A is an illustration showing a tentative reconstructed image, FIG. 8Bis an illustration showing an operation for choosing an object to bemodified, FIG. 8C is an illustration showing an operation for fixing anobject to be modified, FIG. 8D is an illustration showing a paralleltranslation operation; FIG. 8E is an illustration showing areconstructed image after the parallel translation; FIG. 8F is anillustration showing a reduction operation, and FIG. 8G is anillustration showing a reconstructed image after the reduction;

FIG. 9A is an illustration showing a tentative reconstructed imageregarding the embodiment 1;

FIG. 9B is an illustration showing a tentative reconstructed depth map;

FIG. 9C is an illustration showing a converted and reconstructed image;

FIG. 9D is an illustration showing a converted and reconstructed depthmap;

FIG. 10 is a flow chart which shows an image output process of theembodiment 1;

FIG. 11 is a flow chart which shows a tentative reconstructed imagegeneration process of the embodiment 1;

FIG. 12 is a flow chart which shows the conversion setting acquisitionprocess of the embodiment 1;

FIG. 13 is a flow chart which shows a converted and reconstructed imagegeneration process of the embodiment 1;

FIG. 14A is a diagram showing a physical configuration of an imagegenerating apparatus of an embodiment 2;

FIG. 14B is a diagram showing a functional configuration of the imagegenerating apparatus;

FIG. 15A to FIG. 15E are illustrations for explaining operations ofinstructing the deletion regarding the embodiment 2, FIG. 15A is anillustration showing a tentative reconstructed image, FIG. 15B is anillustration showing an operation for choosing an object to be deleted,FIG. 15C is an illustration showing an operation for fixing an object tobe deleted, FIG. 15D is an illustration showing the reconstructed imagein which the object has been completely deleted, and FIG. 15E is anillustration showing the reconstructed image in which a certainpercentage of the object has been deleted;

FIG. 16A to FIG. 16D are illustrations for explaining the reconstructedimage and reconstructed depth map of the embodiment 2, FIG. 16A is anillustration showing a tentative reconstructed image, FIG. 16B is anillustration showing a tentative reconstructed depth map, FIG. 16C is anillustration showing a deleted and reconstructed image, and FIG. 16D isan illustration showing a deleted and reconstructed depth map;

FIG. 17A to FIG. 17C are figures for explaining the process whichchooses the object to be deleted of the embodiment 2, FIG. 17A is afigure showing a deletion operation, FIG. 17B is a list showing aselection pixel list used for selection, and FIG. 17C is a figureshowing a selection result;

FIG. 18A to FIG. 18D are illustrations for explaining a deletion processof the embodiment 2, FIG. 18A is an illustration showing a light fieldimage LFI before deletions;

FIG. 18B is an illustration showing a tentative reconstructed image,FIG. 18C is an illustration showing a light field image LFI afterdeletions, and FIG. 18D is an illustration showing a deleted andreconstructed image;

FIG. 19 is a flow chart which shows an image output process according tothe embodiment 2;

FIG. 20 is a flow chart which shows a tentative reconstructed imagegeneration process according to the embodiment 2;

FIG. 21 is a flow chart which shows a deletion object defining processaccording to the embodiment 2;

FIG. 22 is a flow chart which shows a deletion depth determining processaccording to the embodiment 2;

FIG. 23 is a flow chart which shows a deletion pixel extraction processaccording to the embodiment 2;

FIG. 24 is a flow chart which shows a deleted and reconstructed imagegeneration process according to the embodiment 2;

FIG. 25 is a list showing an example of a class-depth corresponding listof another embodiment of the present invention; and

FIG. 26 is a list showing an example of a selection pixel list ofanother embodiment of the present invention.

DETAILED DESCRIPTION

A digital camera and an image generating apparatus (reconstructed imagegenerating apparatus) according to an embodiment of the presentinvention is described hereinafter with reference to the attacheddrawings. In addition, the similar symbols are given to similar orcorresponding portions in the drawings.

(Embodiment 1)

The embodiment 1 of the present invention is described.

The image generating apparatus 30 (reconstructed image generatingapparatus) according to the embodiment of the present invention isinstalled in a digital camera 1 as shown in FIG. 1.

The digital camera 1 has the following functions (i-vi):

i) a function which shoots a light field image which includes aplurality of sub images shot an object from a plurality of viewpoints;

ii) a function of acquiring a depth coefficient which shows a depth ofan object;

iii) a function generating a reconstructed image in which the image ofthe object is reconstructed from the light field image;

iv) a function which displays the generated reconstructed image;

v) a function which accepts an operation instructing a compositionmodification of the reconstructed image; and

vi) a function of generating a reconstructed image whose composition hasbeen changed according to the accepted operation.

The image generating apparatus 30 takes charge of, especially, “vi) afunction of generating a reconstructed image whose composition has beenchanged according to the accepted operation”.

The digital camera 1 comprises an imager 10, an information processor 20including the image generating apparatus 30, a storage 40, and aninterface (I/F) 50, as shown in FIG. 1. The digital camera 1, with thedescribed configuration, acquires light beam information of an objectfrom outside, and modifies the composition of the object to display.

The imager 10 comprises an optical device 110 and an image sensor 120,and performs an imaging operation.

The optical device 110 comprises a shutter 111, a main lens ML, and asub lens array SLA (micro lens array), as shown in FIG. 2. The opticaldevice 110 receives the light beam from outside (object) through themain lens ML, and projects optical images, which are obtained throughoptical centers of each sub lens SL which constructs sub lens array SLAas viewpoints onto the image sensor 120.

The image sensor 120 comprises, for example, an imaging element, such asa Charge Coupled Device (CCD) or a Complementary Metal OxideSemiconductor (CMOS), and a transmitter which transmits an electricalsignal generated by the imaging element to an information processor 20.The image sensor 120 having the above physical configuration convertsthe optical image projected by the optical device 110 into an electricalsignal and transmits it to the information processor 20.

The shutter 111 controls the incidence and shut of the external light tothe image sensor 120.

The main lens ML comprises a convex lens, a concave lens, an asphericlens, and the like, or a plurality of those lenses and forms an opticalimage using the light from the object OB at the time of photo shootingon a virtual main lens imaging plane MIP between the main lens ML andsub lens array SLA. In addition, the object OB at the time of photoshooting is presupposed to include a plurality of components which areaway from the main lens ML by different distances respectively as shownin FIG. 2.

The sub lens array SLA includes M×N sub lenses (micro lens) SL arrangedin the shape of a grid on a plane. The sub lens array SLA forms opticalimages which are formed on the main lens imaging plane MIP by the mainlens ML as the optical images observed from the optical centers of eachsub lens SL as viewpoints on an imaging plane IE of image sensors whichconstructs the image sensor 120. A space defined by a plane by the mainlens ML and an imaging plane IE is referred to as a light field.

A maximum lens diameter LD and an effective diameter ED can be definedfor the main lens ML. The maximum lens diameter LD is a physicaldiameter of the main lens ML. On the other hand, the effective diameterED is a diameter of an area of the main lens ML which can be used forphoto shooting. The outside of the effective diameter ED of the mainlens ML is an area (ineffective area) which is not effective for photoshooting and/or restructuring images, as the incoming or outgoing lightbeams from or to the main lens are limited by various filters attachedto the main lens ML and/or the physical structure of the circumferenceof the main lens ML.

The maximum lens diameters LD and the effective diameters ED aremeasured in advance, and are stored by a storage 40 set to factorydefault.

In an example in FIG. 2, among a plurality of objects OBs (objectOB1˜object OB3), the light beam from a portion POB of an object OB2passes the part (effective part) which forms the effective diameter EDof the main lens ML, and is projected onto a plurality of sub lensesSLs. In this manner, an area in which the light emitted from the portionPOB of a certain object OB passes the effective part of the main lens MLand is projected onto the sub lens array SLA is referred to as a mainlens blur MLB of the portion POB. Among these, a portion at which a mainlight beam reaches is referred to as the main lens blur center MLBC.

In addition, hereinafter, a plurality of objects are described as anobject OB1 through an object OB3 sequentially in order from the mostdistant (the distance from the main lens is largest) object.

The distance from the optical center of the main lens ML to the imagingplane of the main lens MIP is set to b1, the distance from the imagingplane MIP to a plane formed by the sub lens array SLA is set to a2, andthe distance from the sub lens array SLA to the imaging plane IE of theimage sensor 120 is set to c2.

The imager 10 with the above-mentioned configuration shoots a lightfield image LFI which includes the information on the light beam (lightbeam reached part, light volume, direction) which passes through thelight field. An example of the light field image LFI which shot anobject OB composed of blocks is shown in FIG. 3A. This light field imageLFI includes images (sub images SI, S₁₁-S_(MN)) corresponding to each ofM×N sub lenses SL (micro lens) arranged in the shape of a grid. Forexample, the upper left sub image S₁₁ corresponds to the image shot theobject OB from the upper left, and the lower right sub image S_(MN)corresponds to the image shot the object OB from the lower right.

Each sub image is arranged at a position on the light field image LFIcorresponding to the position of the sub lens which formed the subimage. The i-th row sub images (a lateral row of sub images) Si1-SiNcorrespond to stereo images in which the images formed by the main lensML are formed by sub lenses SL which are arrayed laterally beside thei-th row of the sub lens array SLA. Similarly, the j-th column subimages (a longitudinal column of sub images) S1 j-SMj correspond tostereo images in which the images formed by the main lens ML are formedby sub lenses SL which are arrayed longitudinally beside the j-th columnof the sub lens array SLA. In addition, in this embodiment, each subimage is a gray scale image, and each pixel which constructs a sub imagehas a pixel value (scalar value).

The information processor 20 shown in FIG. 1 comprises a Central ProcessUnit (CPU), a Random Access Memory (RAM), an internal bus, and an I/Oport physically. With the above physical configuration, the informationprocessor 20 functions as an image processor 210, a depth estimation andcorrespondence determination unit 220, the image generating apparatus30, and an image controller 230.

The image processor 210 acquires an electrical signal from the imagesensor 120 and converts the acquired electrical signal to image databased on image setting information stored by an image setting storage410 of the storage 40. The image processor 210 transmits image data andshooting setting information generated by adding predeterminedinformation to the image setting information, to the depth estimationand correspondence determination unit 220.

The image setting information stored by the image setting storage 410 ismentioned later.

The depth estimation and correspondence determination unit 220 estimatesthe depth of an object using the depth of each pixel in the sub image ina light field image LFI, when the light field image LFI and shootingsetting information are received. The depth estimation andcorrespondence determination unit 220 calculates a depth of an objectfrom a deviation magnitude of the pixels corresponding to the object foreach sub image, and the deviation magnitude is set to a coefficientindicating the depth of the object. The depth estimation andcorrespondence determination unit 220 generates information in which thecoefficients indicating the estimated depth are positioned at each pixelon the light field depth map LFDM. It is estimated that, the larger thedepth coefficient is, the closer the object imaged in the pixel ispositioned.

The light field depth map LFDM defines coefficients indicating the depthfor each sub-pixels included in the light field image LFI. A light fielddepth map LFDM corresponding to the light field image LFI shown in FIG.3B is shown in FIG. 3C. In the light field depth map LFDM map, aportion, which is in a more distant position (distant side of therectangular parallelepiped) with regard to the light field image LFIimaging the rectangular parallelepiped object as shown in FIG. 3B, isindicated in a darker color, and a portion which is in a less distantposition (front of the rectangular parallelepiped) is indicated in alighter color. Hereinafter, similarly, in a depth map, a portion whichis in a more distant position is indicated in a darker color, and aportion which is in a less distant position is indicated in a lightercolor.

In addition, although the light field image LFI and the light fielddepth map LFDM are explained as separate pieces of information, thelight field image LFI and the light field depth map LFDM may be oneintegrated graphic in which a pixel value (information held by the lightfield image LFI) and a depth coefficient (information contained in thelight field depth map LFDM) are associated and imaged for a pixelpositioned at the certain coordinates.

Although the light field depth map LFDM can be generated using anymethod which estimates the depth of each pixel of a multiple viewpointimage, the below described method is used in this embodiment.

The depth estimation and correspondence determination unit 220 furtheracquires corresponding information which determines the correspondencerelationship between pixels (pixels to be reconstructed) which constructthe reconstructed image before the composition modification (tentativereconstructed image) and sub-pixels. Corresponding information is theinformation which shows the sub-pixels (corresponding sub-pixels) inwhich an object at the position of the reconstructed image is arrangedon the reconstruction plane indicated by a reconstruction setting.

The depth estimation and correspondence determination unit 220 maygenerate corresponding information by any known method which extractscorresponding sub-pixels of the pixels to be reconstructed (markedpixels). However, in this embodiment, the corresponding sub-pixels areextracted by the following light beam trace.

The method of light beam trace is described with reference to FIG. 4.The light beam from a marked portion P of an object (corresponds to amarked pixel) passes the principal point of the main lens, and reachesto a position to be reached of the micro lens array (main lens blurcenter MLBC on the sub lens in FIG. 4). The position on the sub lens ofthe main lens blur center MLBC can be acquired based on a photo shootingsetting. An area, in which the light from a marked portion reachedcentering around the main lens blur center MLBC (main lens blur MLB,hatched area of FIG. 4), is acquired from the characteristic of thelenses. The diameter of the main lens blur MLB is calculated from adistance a1 between the main lens ML and the reconstruction plane RF, adistance b1 between the main lens and the imaging plane MIP (calculatedfrom the distance a1 and the focal distance f_(ML) of the main lens), adistance a2 between the imaging plane MIP and the sub lens array SLA,and an effective diameter ED of the main lens using triangularsimilarity.

Then, a sub lens SL, totally or partially included in the main lens blurMLB among the sub lenses SL included in the sub lens array SLA, isspecified. The specified sub lens SL is sequentially chosen as a markedlens. An area w of a portion in which the main lens blur MLB overlapswith the marked lens is acquired from the position of the MLBC main lensblur, the diameter of the main lens blur MLB, and the position and sizeof the sub lens determined by the shooting setting information.

A pixel on a sub image (corresponding pixel) is extracted, the pixel isarranged at a position at which the light beam from a marked pixel isformed by the selected sub lens.

Specifically, a corresponding pixel (corresponds to the reaching pointPE) is calculated in the following procedure.

First, the distance b1 to the focal plane of the main lens correspondingto the reconstruction plane RF can be calculated from the followingformula (1) using the known distance a1 and the focal distance f_(ML).

$\begin{matrix}{{b\; 1} = \frac{a\; 1 \times f_{ML}}{{a\; 1} - f_{ML}}} & (1)\end{matrix}$

Moreover, the distance a2 can be acquired by subtracting the distance b1which is calculated using the formula (1) from the known distance c1.

Furthermore, using the distance a1 between the reconstruction plane RFand the main lens, distance b1 between the main lens and the imagingplane MIP, and a known distance x (distance between the marked portion Pand an optical axis OA) using the following formulas (2), distance x′,between the optical axis OA and an image forming point (imaging pointPF) at which the marked portion P from the optical axis OA through themain lens ML forms an image, is calculated.

$\begin{matrix}{x^{\prime} = {x \times \frac{b\; 1}{a\; 1}}} & (2)\end{matrix}$

Furthermore, a distance x″ between the reaching point PE and the opticalaxis OA is calculated by applying, a distance d from the optical axis OAto the principal point of the marked sub lens SL, the distance x′calculated by using the above-mentioned formula (2), the distance c2from the micro lens array LA to the imaging plane IE and the distancea2, to the following formula (3).

$\begin{matrix}{x^{\prime\prime} = {{\left( {d - x^{\prime}} \right) \times \frac{c\; 2}{a\; 2}} + d}} & (3)\end{matrix}$

The reaching point PE is also calculated in the direction of Y-axis in asimilar manner and specified. Pixels on the sub image corresponding tothe reaching point PE are set to corresponding pixels. The area wacquired for the marked lens is set to be weights for the marked pixeland the corresponding pixel.

A corresponding list which is an example of corresponding information isshown in FIG. 5. The corresponding list of FIG. 5 associates and recordsthe coordinates (xr, yr) on the reconstructed image of the pixels to bereconstructed, coordinates (xs, ys) and weights (w) on the light fieldimage LFI of the sub-pixels extracted by the light beam trace for thepixels to be reconstructed of the coordinates. A plurality of sub-pixelsare extracted for one pixel to be reconstructed. For example, in theexample of FIG. 5, the k₁ sub-pixels located at (xs (1, 1), ys (1,1))˜(xs (1, k₁), ys (1, k₁)) are extracted for a first pixel to bereconstructed (coordinates (1, 1)). The strengths (weights) ofcorrespondence of the extracted sub-pixels and pixels to bereconstructed are W (1, 1)˜W(1, k₁).

The depth estimation and correspondence determination unit 220 transmitssuch corresponding information, the light field image LFI and the lightfield depth map LFDM to the image generating apparatus 30.

The image generating apparatus 30 generates an image in which thecomposition of an object is converted and reconstructed (a converted andreconstructed image) using the light field image LFI and the light fielddepth map LFDM transmitted from the image processor 210.

Moreover, the image generating apparatus 30 outputs the generatedconverted and reconstructed image to the image storage 430 of thestorage 40. The details of the configuration of the image generatingapparatus 30 and a process which generates the converted andreconstructed image are described later.

The image controller 230 controls the imager 10 based on the imagesetting information stored in the image setting storage 410 of thestorage 40, and photo shoots an object OB using the imager 10.

The storage 40 includes a main storage which includes RAM and the like,and an external storage which includes a non-volatile memory such as aflash memory, a hard disk and the like.

The main storage loads a control program and information which arestored in the external storage, and is used as the workspace of theinformation processor 20.

In advance, an external storage stores the control program andinformation which allow the information processor 20 to perform thefollowing process, and transmits the control program and information tothe main storage according to instructions of the information processor20. The external storage stores the information based on the process ofthe information processor 20 and the information transmitted from theinterface 50 according to the instructions of the information processor20.

Functionally, the storage 40 includes an image setting storage 410, areconstruction setting storage 420, and an image storage 430.

The image setting storage 410 stores image setting information. Theimage setting information, as an imaging parameter which may be variedat the time of imaging, includes the distance between the main lens MLand the sub lens array SLA, the focal distance f_(ML) of the main lens,information specifying exposure time, F value, a shutter speed, and thelike. Moreover, the image setting storage 410 stores informationregarding the physical configuration of the digital camera 1 such as thepositions of each sub lens SLs on the sub lens array SLA, the distancec2 between the sub lens array SLA and the imaging plane IE and the like.

The image setting storage 410 transmits an imaging parameter to theimage controller 230.

Moreover, the image setting storage 410 adds the image settinginformation of the light field image LFI imaged by the imager 10 to theinformation regarding the physical configuration, and transmits to theimage processor 210 as shooting setting information.

The reconstruction setting storage 420 stores a setting parameter forgenerating an image reconstructed from the light field image LFI. Thesetting parameter is set as a default value or inputted by the userusing the operator 530.

The image storage 430 stores an output image generated by the imagegenerating apparatus 30. The image storage 430 transmits a stored imageto the I/O device 510 and the display 520 of the interface 50.

The interface (described as an I/F part in the diagram) 50 is aconfiguration of an interface interfacing the digital camera 1 with theuser, or with an external device, and includes the I/O device 510, thedisplay 520, and the operator 530.

The I/O device (input/output device) 510 physically includes a UniversalSerial Bus (USB) connector, a video output terminal, and an input-outputcontroller. The I/O device 510 outputs the information stored by thestorage 40 to an external computer, and transmits the informationtransmitted from outside to the storage 40.

The display 520 includes a liquid crystal display, an organic ElectroLuminescence (EL) display and the like, and displays the screen forinputting the imaging parameter stored by the image setting storage 410,and the screen for operating the digital camera 1. Moreover, the display520 displays an image stored by the image storage 430.

The operator 530 includes a transmitter which detects informationregarding, for example, various buttons disposed in the digital camera1, a touch panel disposed on the display 520 and the operationsperformed on the various buttons and/or the touch panel, and transmitsthe information regarding the user operation to the storage 40 and theinformation processor 20.

Then, the configuration of the image generating apparatus 30 isdescribed with reference to FIG. 6A and FIG. 6B.

The image generating apparatus 30 physically includes an informationprocessor 31, a main storage 32, an external storage 33, an input andoutput device 36, and an internal bus 37 as shown in FIG. 6A.

The information processor 31 includes a Central Process Unit (CPU) and aRandom Access Memory (RAM).

The main storage 32 has a physical configuration similar to the mainstorage of the storage 40. The external storage 33 has a physicalconfiguration similar to the external storage of the storage 40 andstores a program 38. The input and output device 36 includes aninput/output terminal and an I/O device, and realizes input and outputof information regarding the image generating apparatus 30, each part ofthe information processor 20, the storage 40, the interface 50 and thelike. The internal bus 37 connects the information processor 31, themain storage 32, the external storage 33, and the input and outputdevice 36.

The information processor 31, the main storage 32, the external storage33, the I/O device 36 and the internal bus 37 may be functional blocksrealized by the internal circuit of the information processor 20 of thedigital camera 1, the storage 40, and the interface 50.

The image generating apparatus 30 copies the program 38 and data storedin the external storage 33 to the main storage 32 and performs a processfor generating the converted and reconstructed image which is mentionedlater through the information processor 31 executing the program 38using the main storage 32.

The image generating apparatus 30 functions as an input device 310, areconstructed image generator 320 and an output device 330 with theabove physical configuration as shown in FIG. 6B.

The input device 310 is a part which takes charge of a function in whichthe image generating apparatus 30 acquires information from each part ofthe digital camera 1. The input device 310 includes a reconstructionsetting acquirer 3110, an LFDM acquirer 3120, an LFI acquirer 3130, acorresponding information acquirer 3140 and a modification operationacquirer 3150.

The reconstruction setting acquirer 3110 acquires reconstruction settinginformation from the reconstruction setting storage 420. The informationreconstruction setting information includes information which describesthe specific details of the reconstruction process and informationnecessary for the image generation process as described below(information indicating a reconstruction parameter and a thresholdvalue). In this embodiment, a reconstruction parameter presupposes togenerate the reconstructed image through the following processes.

-   i) A plurality of layer reconstructed images reconstructed using the    pixels, each of the pixels has a depth coefficient within a    predetermined range, is generated once. A tentative reconstructed    image is generated by superposing each of layer reconstructed    images.-   ii) A tentative reconstructed image is displayed and a user's    composition modification operation is accepted.-   iii) An image whose composition has been changed is reconstructed    and output based on the composition modification operation.

For this reason, the reconstruction parameter includes information whichspecifies the distance between the focal point of a new image and themain lens ML (distance to be reconstructed, distance a1), a range of thedepth coefficients corresponding to each layer, and the like.

The LFDM acquirer 3120 acquires the light field depth map LFDM from thedepth estimation and correspondence determination unit 220.

The LFI acquirer 3130 acquires the light field image LFI generated bythe image processor 210.

The corresponding information acquirer 3140 acquires correspondinginformation (corresponding list of FIG. 5) from the depth estimation andcorrespondence determination unit 220.

The modification operation acquirer 3150 receives, from the operator530, the information regarding an operation instructing a compositionmodification of the reconstructed image from the operator 530.

Each part of the input device 310 transmits the acquired information toeach part of the reconstructed image generator 320.

According to the reconstruction setting information acquired by thereconstruction setting acquirer 3110 and the modification operationinformation acquired by the modification operation acquirer 3150, thereconstructed image generator 320 generates the reconstructed image inwhich the composition of an object has been changed, from the lightfield depth map LFDM acquired by the LFDM acquirer 3120, the light fieldimage LFI acquired by the LFI acquirer 3130 acquired and thecorresponding information acquired by the corresponding informationacquirer 3140.

In the present embodiment, a tentative reconstructed image which is aninitial reconstructed image whose composition has not been changed, isgenerated once, and is then output. When a user performs a compositionmodification operation using the tentative reconstructed image, theconverted and reconstructed image which is a reconstructed image whosecomposition has been changed is generated. In conjunction with thegeneration of the converted and reconstructed image, a reconstructeddepth map which defines the depths of each pixel is also generated.

For this reason, the reconstructed image generator 320 includes a layerdeterminer 3210, a layer image generator 3220, a tentative reconstructedimage generator 3230, a conversion pixel extractor 3240, a conversionmatrix determiner 3250, a converter 3260, and a converted andreconstructed image generator 3270.

The layer determiner 3210 defines the layers of a reconstructed image(both for a tentative reconstructed image and a converted andreconstructed image). That is, the images corresponding to the depths(depth coefficients) within a predetermined range are allocated to eachlayer, setting the reconstructed image as an image on which a pluralityof layers are superimposed. Layer images having the same resolution asthe (tentative) reconstructed image are placed on each layer. The numberof layers defined by the layer determiner 3210 and the depthcoefficients allocated to each layer are determined by thereconstruction setting.

In the present embodiment, three layers are defined and the assumabledepth coefficient in a design is divided into three, and allocated tothe three layers respectively. Among the three layers, the most distantlayer is referred to as a layer 1, the middle distant layer is referredto as a layer 2 and the closest layer is referred to as a layer 3.

The layer determiner 3210 transmits the information regarding thedefined layer to the layer image generator 3220.

The layer image generator 3220 generates images of each layer (layerimages) as follows:

-   (1) One of the pixels constructing a layer image (layer pixels) is    set to a marked pixel and corresponding sub-pixels of the pixel to    be reconstructed which corresponds to the coordinates of the marked    pixel are acquired with reference to the corresponding information,    and are set as corresponding pixel candidates.-   (2) The depth coefficient of the corresponding pixel candidate is    acquired with reference to the light field depth map LFDM.-   (3) Among the corresponding pixel candidates, a pixel, which is    within the range in which a depth coefficient is allocated to the    layer, is extracted as the corresponding pixel of the marked pixel    and its pixel value is acquired.-   (4) An acquired pixel value multiplied by a weighting coefficient is    set to be a corrected pixel value.-   (5) Corrected pixel values are calculated and summed for all the    extracted corresponding pixels.-   (6) The sum of the corrected pixel values is divided by the sum of    the overlap areas (sum of weights) to be the pixel value of the    marked pixel. When there is no corresponding pixel (or less than a    predetermined threshold value), the pixel value is set to NULL.-   (7) The above processes (1)-(6) are performed for each layer pixel    of each layer, and pixel values are determined.

The layer image generator 3220 further generates the depth map of eachlayer image. Specifically, the layer image generator 3220 sets the depthcoefficient of the light field depth map LFDM as a pixel value, extractsa corresponding pixel in a similar manner as the layer image, and thecorresponding pixel is weighted and added in order to set the depthcoefficient of each layer pixel. Or, among the depth coefficients of thecorresponding pixels, a mode may be a depth value of the pixels to bereconstructed.

The layer image generator 3220 transmits the generated layer image andlayer depth map to the tentative reconstructed image generator 3230.

The tentative reconstructed image generator 3230 generates a tentativereconstructed image (the reconstructed image before the compositionmodification, RI1) by superimposing layers. Moreover, a reconstructeddepth map (RDM1) is generated from the layer depth map.

This process is described with reference to FIG. 7A through FIG. 7C. Theimager 10, as shown in FIG. 7A, is assumed to shoot images a distantobject OB1 (dashed line), object OB2 (alternate long and short dashline) in a medium position, and a near object OB3 (solid line). Theobject OB1 is assumed to be positioned in the range indicated by thedepth coefficient allocated to the layer 1 (the distance from the mainlens ML is within a predetermined range). The object OB2 and object OB3are assumed to be positioned in the range indicated by the depthcoefficients allocated to the layers 2 and 3, respectively in a similarmanner.

In this case, from the light field image LFI and the light field depthmap LFDM, the layer image generator 3220 generates a layer image 1 (L1)for the layer 1, a layer image 2 (L2) for the layer 2, and a layer image3 (L3) for the layer 3, respectively (FIG. 7B). In each layer image, thepixel value of the layer pixel which does not have an object in thecorresponding depth range is NULL, and when superimposed, the pixelvalue is to be processed through the transmissive process.

The tentative reconstructed image generator 3230 sets the pixel value ofthe pixel which constructs the tentative reconstructed image (tentativepixels to be reconstructed) to the pixel value of the closest layerwhose pixel value is not NULL among the pixels of each layer of the samecoordinates. This process is performed for all the tentative pixels tobe reconstructed to generate the tentative reconstructed image (RDM1).This process is referred to as superimposition process. A reconstructeddepth map (RDM1) is generated by superimposing layer depth maps (FIG.7C) in a similar manner.

In this case, an object hidden by the object of the layer correspondingto a closer position (for example, layer 2) is imaged on a layercorresponding to a position distant from the main lens ML (for example,layer 1) when superimposed without any composition modification. In theexample of FIG. 7B, the object OB1′ is imaged on the layer 1. However,as the object OB2 exists on the layer 2 corresponding to a closerposition, the OB1 is not imaged in the tentative reconstructed image(RI1). Although the object OB1′ is visible from some sub lenses(viewpoint at which a certain sub image is shot), the object is notvisible from other sub lenses. Thus, a phenomenon, in which an objectwhich is imaged in a certain sub image but not imaged in other subimages is generated due to the differences of the viewpoints, isreferred to as an occlusion. Moreover, the sub-pixels corresponding tosuch an object is referred to as occlusion pixels.

As the light field image LFI is constructed from a plurality ofsub-pixels respectively shot from different viewpoints, occlusion pixelsare included. When shot from one viewpoint, the object corresponding toocclusion pixels cannot acquire the information hidden by a closerobject. The digital camera 1 according to the present embodiment shootssuch a hidden object by photo shooting the object from a plurality ofviewpoints (corresponding to each sub lens), and forms an image onto adistant layer. This image is used in order to generate a reconstructedimage whose composition has been changed.

The tentative reconstructed image generator 3230 transmits the generatedtentative reconstructed image (RI1) to the output device 330 to output.Moreover, the tentative reconstructed image generator 3230 transmits thetentative reconstructed image (RI1) and the reconstructed depth map(RDM1) to the conversion pixel extractor 3240. The output device 330stores the tentative reconstructed image (RI1) in the image storage 430to display on the display 520.

The display 520 shows a user a tentative reconstructed image (RI1), andthe operator 530 accepts an operation for choosing an object on thetentative reconstructed image (RI1) and an operation which modifies theposition and size of the chosen object (composition modificationoperation) from the user. The modification operation acquirer 3150acquires the received operations. The conversion pixel extractor 3240extracts an area occupied by the object whose composition is to bemodified in each layer (the layer in which the image of the objectexists, and the layer pixels) based on the operations acquired by themodification operation acquirer 3150.

Operation reception process is described with reference to the examplesof FIG. 8A through FIG. 8G. It assumes that a tentative reconstructedimage (RI1) contains the (white) background, a (gray) child that existsnear the focal plane RF, and a (black) obstacle close to the child asshown in FIG. 8A. The user who observed the tentative reconstructedimage (RI1) displayed by the display 520 displays desires to move or toreduce the obstacle close to the child in order not to block the imageof the child in viewing the image. On this screen, the user firstchooses an object by an operation such as tracing the obstacle and thelike on a touch panel (the display 520 and the operator 530) as shown inFIG. 8B. The conversion pixel extractor 3240 extracts the layer pixelscorresponding to the object selected based on this operation (that is,the corresponding sub-pixels) as pixels to be converted. In addition, awhite arrow shows an operation with the finger on a touch panel. Theprocess which extracts pixels to be converted from this operation ismentioned later.

The display 520 highlights and displays the pixels to be reconstructedcorresponding to the pixels to be converted and allows the user tochoose whether or not the selected object is correct using a dialog box(FIG. 8C). If the user determines that the selected object is a mistake,the user touches the “NO” portion. In this case, the selection is resetand the digital camera 1 awaits a new selection operation. On the otherhand, when the present selection is correct, the user touches the “YES”portion to fix an object to be converted (corresponds to the pixel to beconverted).

When the pixel to be converted is fixed, then, the user performs anoperation defining conversion parameter. First, the user chooses how thecomposition of an object to be converted is modified using the operator530. Suppose that, in the present embodiment, the user chooses one ofmovement, rotation and expansion/reduction functions.

For example, when a movement is chosen, the user performs an operationof specifying a moving direction and a moving distance. FIG. 8D showsthe operation which moves an object to be converted (obstacle) in thespecified quantity in the left direction on a touch panel as an exampleof such an operation. In response to such an operation, the imagegenerating apparatus 30 generates the reconstructed image (converted andreconstructed image) in which the obstacle is moved as shown in FIG. 8E.

Moreover, when a reduction is chosen, the operation of specifying thedirection of reduction (length or width) and the grade of reduction isperformed. FIG. 8F shows an example of the operation which reduces theselection area in a lateral direction by the operation of pinching theselection area with two fingers. In response to the above operation, theimage generating apparatus 30 generates a reconstructed image (convertedand reconstructed image) in which the obstacle is reduced as shown inFIG. 8G

The user can further instruct a composition modification on thereconstructed image whose composition has been changed as shown in FIG.8E or FIG. 8G and repeat composition modifications until an image of adesired composition is obtained.

In addition, this operation is not limited to the operation on a touchpanel, and, for example, parameters such as a moving distance and thelike may directly be input through a dialog box.

The conversion matrix determiner 3250 defines the parameter for theconversion (conversion matrix) from the modification operation acquiredby the modification operation acquirer 3150.

The affine transformation matrix in the formula (4) is described as anexample of a conversion matrix.

$\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime}\end{pmatrix} = {\begin{bmatrix}{a\; 1} & {b\; 1} & {d\; 1} \\{a\; 2} & {b\; 2} & {d\; 2}\end{bmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}} & (4)\end{matrix}$

In addition, x and y are coordinates in a conversion layer of the pixelsto be converted before the conversion, and x′ and y′ are the coordinatesafter the conversion. Layer pixels are the pixels which associate pixelsto be reconstructed and sub-pixels for each depth. Therefore, convertingthe coordinates of the layer pixels is equal to changing thecorrespondence relationship between the pixels to be reconstructed andthe sub-pixels.

Each element of the affine transformation (conversion matrix) can beacquired by a general method which determines each element of the affinetransformation matrix from the conversion method and its parameter. Forexample, when moved parallel by x1 pixels in the direction of x axis, a1and b2 are set to 1, d1 is set to x1, and b1, a2 and d2 are set to 0.When reduced to ½ in the y direction, a1 is set to 1, b2 is set to ½,and d1, b1, a2 and d2 are set to 0. When rotated by an angle θ, a1 andb2 are set to cos θ, b1 is set to minus sin θ and a2 is set to sin θ.

In addition, the center of expansion, reduction, and rotation can befreely set by carrying out coordinate conversion of the coordinates of(x, y) and (x′, y′) in advance such that the origin is set to a desiredcenter. In the present embodiment, the weighted center of the area to beconverted is defined, and supposes that affine transformation isperformed using the formula (4) after performing a coordinate conversionso that rotation, expansion, and reduction can be performed focusing onthe weighted center.

Specifically, suppose that the coordinates of the weighted center are(xc, yc) and the coordinates of the pixel to be converted are (x1, y1).(x′, y′) are calculated by setting, (x, y)=(x1−xc, y1−yc), using theformula (4). After the calculation, xc is added to x′ and yc is added toy′ to be the final calculation result. In addition, “the weightedcenter” here is the result of an average of the summed coordinate valuesof all the pixels to be converted.

In addition, the method of determining each element of the matrix ofaffine transformation can be substituted for any known method. Forexample, it is possible to use a method in which a user specifies threepoints in the selection area and the moved portion of the three points,and generate a matrix based on the moving vector of the points.

Moreover, a user may directly input each element of the determinant. Theconversion matrix determiner 3250 transmits the information indicatingthe defined matrix to the converter 3260.

The converter 3260 modifies a composition of an object on the layer(conversion layer) in which a conversion area exists using theconversion matrix transmitted from the conversion matrix determiner3250. The depth map of the conversion layer is also converted in asimilar manner accompanied by the conversion of the composition of theobject. The converter 3260 may perform a composition conversion in anyknown method in which a composition of an image is converted using aconversion matrix. However, specifically, composition conversions arerealized by the process mentioned later.

The converter 3260 transmits to the converted and reconstructed imagegenerator 3270 the converted layer image and the depth map.

The converted and reconstructed image generator 3270 determines thepixel values of each pixel of the converted and reconstructed image(RI2) whose composition has been converted, and generates a convertedand reconstructed image. The converted and reconstructed image generator3270 generates the depth map corresponding to a converted andreconstructed image (RI2) (a converted and reconstructed depth map,RDM2) in a similar manner. Specifically each layer image transmittedfrom the converter 3260 is superimposed to determine the pixel values ofeach pixels to be reconstructedimilar to the tentative reconstructedimage generator 3230. A converted and reconstructed depth map (RDM2) isgenerated for a depth coefficient by being superimposed in a similarmanner. Examples of the converted and reconstructed image (RI2) and theconverted and reconstructed depth map (RDM2) which were generatedthrough the above process are described with reference to FIG. 9Athrough FIG. 9D.

FIG. 9A shows a tentative reconstructed image (RI1). FIG. 9B shows thereconstructed depth map (RDM1) of a tentative reconstructed image (RI1).The parts (nearby object) indicated by solid lines are (white) partswith a large depth coefficient on a reconstructed depth map (RDM1).These parts correspond to the layer 3 (closest layer).

On the other hand, the part (more distant object) indicated by thealternate long and short dash line corresponds to the layer 2. This partis indicated in gray on the reconstructed depth map (RDM1). The pixel ofthe same part of the layer 3 has a NULL value because there are nocorresponding sub-pixels.

The parts of an object (background) indicated by dashed lines are shownin a darker color in the reconstructed depth map (RDM1). In these parts,as the pixels of the layer 3 and the layer 2 have NULL values, thepixels of the layer 1 appear in the pixels to be reconstructed.

Suppose that a user performs an operation of moving the object of thelayer 2 to the left. In response to this operation, an image in whichthe object of the layer 2 generated by the image generating apparatus 30is moved to the left is a converted and reconstructed image (RI2) ofFIG. 9C.

With the movement of the object of the layer 2, the gray partcorresponding to the layer 2 is also moved to the left in the convertedand reconstructed depth map (RDM2, FIG. 9D). Here, the source part(before the movement) is complemented with the pixel values of thepixels corresponding to a more distant object (pixels of the layer 1).As a result, the object OB1′ of the layer 1 appears on a converted andreconstructed image (RI2).

Moreover, when a closer object (0B3) exists in the destination for themovement of an object (0B2), the moved object (0B2) is hidden by acloser object (0B3).

Specifically, when the pixels corresponding to the layer whichcorresponds to a near position (layer 3) do not have NULL values, thepixel value of the converted and reconstructed image (RI2) of the partis overwritten with the pixel value of the layer corresponding to thenear position (layer 3) by a superimposition process, and the pixelscorresponding to a more distant layer (layer 2) do not appear on theconverted and reconstructed image (RI2).

As the layer depth map is also modified, in the converted andreconstructed depth map (RDM2), the originally positioned part is darker(having a smaller depth coefficient) than before the movement. Moreover,pixels having larger depth coefficients are given priority in thedestination part of the movement (after movement). As a result, if thereare pixels having larger depth coefficients in the destination part ofthe movement, the pixels corresponding to the moved object are hidden bythe pixels having larger depth coefficients, and if there are pixelshaving smaller depth coefficients, the pixel values are overwritten.

The converted and reconstructed image generator 3270 determines thepixel values of each pixel of the converted and reconstructed image togenerate the converted and reconstructed image in this way, and outputsto the output device 330 the generated converted and reconstructed image(RI2).

The output device 330 stores the converted and reconstructed image (RI2)transmitted from the converted and reconstructed image generator 3270 tothe image storage 430. Then, the display 520 displays the converted andreconstructed image (RI2).

Then, a process which the digital camera 1 performs is described withreference to a flow chart. The digital camera 1 starts the image outputprocess shown in FIG. 10, when a user photo shoots an object using theimager 10.

In the image output process, the image processor 210 first generates thelight field image LFI from the image information acquired from theimager 10, and the shooting setting information acquired from the imagesetting storage 410 (Step S101). Then, the generated light field imageLFI and the shooting setting information are transmitted to the depthestimation and correspondence determination unit 220.

Then, the depth estimation and correspondence determination unit 220acquires shooting setting information. Furthermore, the depth estimationand correspondence determination unit 220 acquires reconstructionsetting information from the reconstruction setting storage 420 (StepS102). The acquired photo shooting setting and reconstruction settingare transmitted also to the image generating apparatus 30.

The depth estimation and correspondence determination unit 220 estimatesthe depth for each sub-pixel which constructs the light field image LFI.Moreover, the image generating apparatus 30 acquires the light fielddepth map LFDM in which the depth coefficients are arranged (Step S103).

Specifically, the depth estimation and correspondence determination unit220 estimates the depth of the corresponding object for each sub-pixelwhich constructs the light field image LFI, and generates a light fielddepth map LFDM which associates the depth coefficients indicating theestimated depth with the sub-pixels.

Here, a depth coefficient may be calculated using any method ofestimating the distance to the object corresponding to the pixels of thelight field image LFI. However, in the present embodiment, depthcoefficients are calculated in the following method:

-   i) A certain sub image which constructs the light field image LFI is    set to be a marked sub image.-   ii) The marked sub image is divided into the image areas constructed    by the pixels whose pixel value differences are included in the    predetermined range. Then, one of the image areas is chosen as a    marked area.-   iii) The sub image on the right-hand side of the marked sub image    (on the left-hand side if there is no sub image on the right-hand    side, in this case henceforth, right and left are reversed) is set    to SR1, SR2, ˜SRk in the positional order. In addition, k is set to    a natural number.-   iv) The coordinates (x, y) of the weighted center of the marked area    in a marked sub image is acquired. In addition, these coordinates    are defined with regard to the independent coordinate systems for    each sub image whose center is the origin.-   v) The present pixel deviation is set to “d”. In the sub image SR1,    the area corresponding to the marked area of the marked sub image    (corresponding area) is arranged. The weighted center of the    corresponding area is deviated by “d” to the right side and placed    at this point. The sum of squares of difference (SSD) between the    pixel values of each pixel of a marked area and the pixel values of    the corresponding pixels of the corresponding area is calculated.    The corresponding area is deviated by “2d” to the right side, placed    in the SR2, and the SSD is calculated in a similar manner. SSDs are    acquired up to SRk, respectively and the absolute value sum of each    SSD (SSSD) is acquired which is set to be the evaluated value of    “d”.-   vi) The evaluated values for each pixel deviation “d” in the range    of possible disparity are calculated. Among them, the pixel    deviation (d) having the minimum absolute difference value sum SSSD    acquired is set to the pixel deviation coefficient of the pixel    included in the marked area.-   vii) Pixel deviation coefficients for all the pixels of all the sub    images are calculated, and the calculated pixel deviation    coefficients are arranged in the parts of corresponding pixels as    depth coefficients of the light field depth map LFDM.

The pixel deviation coefficients for each pixel can determine thepositions in the other images of the pixels corresponding to the markedportion of a certain image, and can be acquired by any known methodwhich calculates the deviation of mutual position. The pixel deviationcoefficient calculated through the above stated process indicates thatthe object corresponding to marked sub-pixels appear as images indifferent positions for each image according to the difference of theviewpoints from which each sub image is shot, and indicates theestimated result of the positional differences (sub image disparity).

Here, the positional difference of the viewpoints for each sub image(sub lens) is referred to as the lens disparity, and the difference ofthe position at which the corresponding images which are generated bythe lens disparity appear is referred to as image disparity. The greaterthe lens disparity is, the greater the image disparity becomes.Moreover, the greater the depth (distance between a viewpoint and anobject) is, the less the image disparity becomes. When the lensdisparity is known in each sub image as in the present embodiment, theimage disparity (deviation coefficient) can be set to the depthcoefficient which shows the estimated depth.

Then, the depth estimation and correspondence determination unit 220generates the corresponding list shown in FIG. 5 by a light beam tracedescribed in FIG. 4 and acquires the corresponding list generated by theimage generating apparatus 30. (Step S104).

After acquiring a light field depth map LFDM at Step S103 and acorresponding list at Step S104, then the image generating apparatus 30starts a process (tentative reconstructed image generation process) thatgenerates a tentative reconstructed image (RI1) using the light fieldimage LFI, the light field depth map LFDM, and a corresponding list(Step S105). Here, a tentative reconstructed image generation process 1shown in FIG. 11 is performed.

In the tentative reconstructed image generation process 1, the layerdeterminer 3210 acquires each layer specified by reconstruction settinginformation first (Step S201). Ranges of the depth coefficientsinstructed by the reconstruction setting acquired by the reconstructionsetting acquirer 3110 are allocated to each layer. Moreover, layerimages having the same resolution as the reconstructed images (RI1 andRI2) are arranged for each layer. For example, in the depth coefficients(pixel deviation coefficients), 0-3 to a layer 1, 4-7 to a layer 2 and8-N (N is a depth coefficient corresponding to the maximum imagedisparity able to be set) to a layer 3 are allocated.

In addition, the number of layers and the depth coefficient allocated toeach layer can be set up freely. Moreover, the layer determiner 3210 maydetermine the ranges of the depth coefficient belonging to each layer byclustering which use the depth coefficients of each pixel of the lightfield depth map LFDM. Any known clustering methods such as a method inwhich k-average algorithm is used and/or the ward method, can be used.

Then, the layer image generator 3220 chooses a marked layer from thelayer defined at Step S201 (Step S202). Furthermore, a marked layerpixel is chosen from the pixel included in a marked layer (layer pixel)(Step S203).

Then, the layer image generator 3220 extracts the corresponding pixelcorresponding to the marked layer pixel from sub-pixels (Step S204).Specifically, in the corresponding list of FIG. 5, the sub-pixelsregistered in a row in which the coordinates of the pixels to bereconstructed match with the coordinates of the marked layer pixels areset to be corresponding candidate pixels. In the corresponding candidatepixels, the pixels included in the range in which the depth coefficientdefined by the light field depth map LFDM is allocated to the markedlayer are extracted as corresponding pixels.

Then, the layer image generator 3220 calculates the pixel value of themarked layer pixel (Step S205). Specifically, the arithmetic weightedmean, of the pixel values of each corresponding pixel extracted at StepS204, is acquired by using the weights of each corresponding pixel. Theacquired value is set to be the pixel value of the marked layer pixel.When the number of corresponding pixels is equal or less than apredetermined threshold value, the pixel value is set to be NULL. Thisthreshold value is defined by reconstruction settings.

The layer image generator 3220 calculates the depth coefficient of amarked layer pixel in a similar manner (Step S206). Specifically, thearithmetic weighted mean, of the depth coefficient defined by the lightfield depth map LFDM for each corresponding pixel extracted at StepS204, is acquired by using the weights of each corresponding pixel. Theacquired value is set to be the depth coefficient of the marked layerpixel.

Then, whether or not the above-mentioned process has been completed isdetermined for all the layer pixels of the marked layer as the markedpixels (Step S207). When there is an unprocessed layer pixel (Step S207;NO), the process is repeated from Step S203 for the next unprocessedlayer pixel. On the other hand, if all the layer pixels have beenprocessed (Step S207; YES), the process is completed for the layer andthe process proceeds to Step S208.

At Step S208, whether or not the above-mentioned process has beencompleted is determined for all the layers defined at Step S201 as themarked layers. When there is an unprocessed layer (Step S208; NO), theprocess is repeated from Step S202 for the next unprocessed layer. Onthe other hand, if all the layers have been processed (Step S208; YES),the generation process of layer images for all layers can be determinedto have completed and the process proceeds to Step S209.

At Step S209, the tentative reconstructed image generator 3230superimposes layer images, and generates a reconstructed image (atentative reconstructed image, RI1). Specifically, the pixelscorresponding to each layer image (corresponding layer pixels) areextracted for each pixel (pixels to be reconstructed) therebyconstructing the tentative reconstructed image (RI1). In thecorresponding layer pixels extracted, the pixel values of the pixelsbelonging to the layer closest to the main lens ML whose pixel value isnot NULL (layer having a large allocated depth coefficient) are set tobe the pixel values of the pixels to be reconstructed. If this processis performed for all the pixels to be reconstructed and pixel values aredetermined, a tentative reconstructed image (RI1) can be generated.Then, the tentative reconstructed image generation process 1 iscompleted.

Returning to FIG. 10, a tentative reconstructed image is generated atStep S105. The reconstructed image generated by the output device 330 isthen output (Step S106), and the output device 330 allows the display520 to display the reconstructed image.

Then, regarding the tentative reconstructed image displayed at StepS106, the image generating apparatus 30 performs a process in which thecomposition modification operation operated by a user is accepted, thepixels to be converted are extracted based on the operation, and themodification parameter is defined (conversion setting acquisitionprocess, conversion setting acquisition process 1 in this case) (StepS107).

The conversion setting acquisition process 1 performed at Step S107 isdescribed with reference to FIG. 12. In the conversion settingacquisition process 1, the modification operation acquirer 3150determines first whether or not the operation for choosing an object tobe converted by a user has been detected (Step S301). Specifically,whether or not the modification operation acquirer 3150 can detect anoperation by the user touching the tentative reconstructed image (RI1)on the touch panel which is constructed with the display 520 and theoperator 530.

When the operation cannot be detected (Step S301; NO), Step S301 isrepeated and the system waits for the detection of the operation.

On the other hand, when a selection operation can be detected (StepS301; YES), the conversion pixel extractor 3240 extracts the layer pixelcorresponding to the selection part (pixels to be selected) (Step S302).Specifically, the coordinates of the pixels to be reconstructed,corresponding to the designated positions through the touch operation tothe touch panel, are acquired. In the pixels positioned at thecoordinates acquired by each layer, the layer pixels of the layerclosest to the main lens ML whose pixel value is not NULL (layer havinga large allocated depth coefficient) is set to be the pixels to beselected.

Then, the conversion pixel extractor 3240 acquires a depth coefficientwith reference to the tentative reconstructed image (RI1) and itsreconstructed depth map (RDM1) (Step S303).

Then, the sequential operation (operation selecting the next part oroperation for completing selection) is awaited. Whether or not the nextdetected operation is the operation for completing selection isdetermined (Step S304). Specifically, whether or not the operationsindicating the completion of selection such as lifting a finger from atouch panel and/or performing an operation instructing the completionare detected, is determined. When such operations are not detected (StepS304; NO), based on an operation for choosing the next part, the processis repeated from Step S302.

On the other hand, when an operation completing the selection isdetected (Step S304; YES), the process proceeds to Step S305. At StepS305, the conversion pixel extractor 3240 extracts a layer including animage of an object to be converted (conversion layer) and pixelscorresponding to an object to be converted (pixels to be converted)based on the pixel value and the depth coefficient acquired at StepS303. Furthermore, as shown in FIG. 8C, the extracted pixels to beconverted are highlighted and displayed.

A method for extracting pixels to be converted can be any known methodin which an area of an image is cut, divided and selected based on auser operation. However, the following method is described.

-   i) All the layers including pixels to be selected are specified. In    the specified layers, a layer which satisfies a predetermined    condition is chosen as a conversion layer. The predetermined    condition can be optionally set from conditions, such as “the number    of the pixels to be selected included in the layer is larger than a    predetermined threshold value”, “the ratio of ‘pixels belonging to    the layer’ to ‘all the pixels to be selected’ is larger than a    predetermined ratio” and the like.-   ii) On a conversion layer, pixels continuous to pixels to be    selected are incorporated in the selection area based on the pixel    values. Specifically, a pixel to be selected on the conversion layer    is set to a first selection area. In the pixels on the conversion    layer adjacent to the current selection area, pixels having pixel    values within the predetermined range are incorporated in the    selection area. The predetermined range is set based on the average    values of the pixel values of the pixels included in the current    selection area. The pixels having NULL values are not incorporated    in the selection area. This incorporation process is repeated until    newly pixels to be incorporated are depleted. The pixels finally    incorporated in the selection area are pixels to be converted.

In addition, the portion occupied by the pixels to be converted isreferred to as conversion area.

Then, whether or not the object to be converted (pixels to be converted)is fixed is determined by the defining operation performed by the userwho confirms the highlighted conversion area (Step S306). When the userchooses NO in FIG. 8C and the like, and the object to be converted isnot fixed (Step S306; NO), the conversion area is reset and the processreturns to the Step S301.

On the other hand, when the user chooses YES in FIG. 8C and the like,and the conversion area is fixed (step S306; YES), the conversion pixelextractor 3240 fixes the pixels included in the conversion area aspixels to be converted and the process proceeds to Step S307.

At Step S307, the modification operation acquirer 3150 determineswhether or not the modification operation as described in FIG. 8D orFIG. 8F is detected. When not detected (Step S307; NO), the systemawaits until the system can detect the modification operation byrepeating the Step S307.

On the other hand, when detected (Step S307; YES), the conversion matrixof the formula (4) is generated based on the detected modificationoperation (Step S308). For example, when the operation reducing in thedirection of the x axis and the operation instructing a reduction ratioare detected, the reduction ratio is substituted for the numerical valueof a1 in the formula (4).

Or when the operation supporting parallel translation and the operationinstructing a moving vector (dx, dy) are detected, dx is substituted ford1 and dy is substituted for d2 in the formula (4) respectively.Moreover, when rotated by an angle θ, a2 and b2 are substituted for cosθ, b1 is substituted for −sin θ and a2 is substituted for sin θ.

If a conversion matrix (affine transformation matrix) is generated atStep S308, the conversion setting acquisition process 1 is completed.

Returning to FIG. 10, if the pixels to be converted are extracted atStep S107 and the conversion parameter is acquired, the converter 3260starts a process which generates the converted and reconstructed imageusing the conversion parameter (converted and reconstructed imagegeneration process, equals to the converted and reconstructed imagegeneration process 1) (Step S108).

The converted and reconstructed image generation process 1 performed atStep S108 is described with reference to FIG. 13. In the converted andreconstructed image generation process 1, first, one of the conversionlayers defined by the conversion setting acquisition process is chosenas a marked layer (Step S401).

Then, the coordinates (xc, yc) of the weighted center of the conversionarea in the marked layer are calculated (Step S402).

Then, one marked pixel to be processed is chosen from the pixels to beconverted on the marked layer extracted in the conversion settingacquisition process. The coordinates, a pixel value, and a depthcoefficient of the marked pixel to be converted are acquired withreference to the layer image and the layer depth map (Step S403). AtStep S403, the coordinates of the marked pixel to be converted arefurther moved to a coordinate system centering on the weighted center.Specifically, (x-xc, y-yc) are calculated by subtracting the coordinatesof the weighted center from the coordinates (x, y) of the acquiredmarked pixel to be converted.

Then, from the coordinates (x−xc, y−yc) of the marked pixel to beconverted acquired by the converter 3260 at Step S403, the converter3260 calculates the coordinates of the conversion destination using theconversion matrix defined by the conversion setting acquisition process(Step S404).

Specifically, the coordinates of the conversion destination (x′, y′) arecalculated by substituting the coordinates (x−xc, y−yc) of the markedpixel to be converted for (x, y) in the formula (4). (x′+xc, y′+yc) areacquired by adding the coordinates of the weighted center and thecoordinate system is returned to the original. (x′+xc, y′+yc) is theconverted coordinates.

Then, the converter 3260 updates the pixel value of the original pixelbefore conversion (Step S405). Specifically, the pixel value of thecoordinates (x, y) of the marked layer before conversion is set to NULL,and, is set to be passed through in the superposition. Also, the depthvalue of the coordinates in the depth map of the marked layer is set toNULL.

Then, the converter 3260 updates the pixel value of the convertedcoordinates (x′+xc, y′+yc) (Step S406). Specifically, the pixel valueacquired at Step S403 is set to the pixel value of the pixel to beconverted. In connection with the above pixel value, the depth value ofthe coordinates in the depth map of the marked layer is set to the depthvalue acquired at Step S403. When the pixel value has already beenmodified in the loops (up to the previous loop) (Step S403˜Step S407)with regard to the pixel to be converted, the pixel value is correctedand updated so that the converted pixel value becomes the average valueof the plurality of the original pixel values before modification.

In addition, it is described that even when the converted pixel value isnot NULL, the pixel value is overwritten by the original pixel valuebefore conversion. However, not being limited to the above example, itmay also be appropriate that when the converted pixel value is NULL, theconverted pixel value is overwritten, and when the converted pixel valueis not NULL, the depth coefficient is compared before and afterconversion and the pixel value of the pixel closer to the main lens MLmay be set to the converted pixel value, or when the pixel value at theconversion destination is not NULL, a message warning that confusion hasoccurred may be displayed and the conversion process may be completed.

Then, it is determined whether or not the above-mentioned process isperformed for all the pixels to be converted on the marked layer (StepS407). If an unprocessed pixel to be converted is on the marked layer(Step S407; NO), the process is repeated from Step S403 for thesubsequent unprocessed pixel to be converted.

On the other hand, if the process is completed for all the pixels to beconverted (Step S407; YES), then whether or not the above-mentionedprocess is completed is determined for all the layers to be converted(Step S408). When there is an unprocessed layer to be converted (StepS408; NO), the process is repeated from Step S401 for the subsequentunprocessed layer to be converted.

On the other hand, when the composition modification process iscompleted for all the layers to be converted, (Step S408; YES), theprocess proceeds to Step S409. At Step S409, the converted andreconstructed image generator 3270 generates a converted andreconstructed image (RI2) by superimposing all the layers. The concretemethod of superposition is similar to Step S209 of the tentativereconstructed image generation process 1 (FIG. 11). When there is apixel whose pixel values for all the layers are NULL by the conversionprocess, a default value (for example, blue) is set to the pixel valueof the pixel. In connection with the above stated pixel value, a warningmessage is displayed which shows that the conversion exceeds the rangein which the pixel value can be defined by a photo shooting image.

Moreover, at Step S409, a converted and reconstructed depth map (RDM2)is further generated in a similar manner in preparation for the casewhere the next composition modification is performed.

Then, the converted and reconstructed image generation process 1 iscompleted.

“Generating the reconstructed image by superposition by performing animage conversion in the converter 3260 for the layer pixel which belongsto the modification layer among the reconstructed images” is equal to“modifying the correspondence relationship between pixels to bereconstructed and sub-pixels only for some pixels corresponding to anobject to be modified”.

In the corresponding information (corresponding list of FIG. 5), thecorresponding sub-pixels are defined for all the layer pixelscorresponding to the position of the object to be modified in thetentative reconstructed image (RI1) before the composition modification.In these sub-pixels, the sub-pixels corresponding to the layer pixels ofthe modification layer are sub-pixels having a depth coefficient in therange allocated to the modification layer.

The conversion pixel extractor 3240 extracts the layer pixels of aconversion layer as pixels to be converted. This equals to “extractingthe sub-pixels having a predetermined depth coefficient in which anobject to be modified is presumed to be imaged, among the sub-pixelscorresponding to the pixels to be reconstructed (converted and pixels tobe reconstructed) at a position of the object to be modified”.

That the converter 3260 converts the position of the pixel to beconverted on a modification layer is substantially equivalent to thatthe sub-pixels corresponding to the layer pixels are set to thecorresponding pixels of the destination modification from thecorresponding pixels of the original pixels to be reconstructed beforemodification.

Returning to FIG. 10, when a converted and reconstructed image isgenerated at Step S108, the converted and reconstructed image generatedby the output device 330 then is output to the image storage 430 (StepS109). The display 520 or the I/O device 510 outputs the converted andreconstructed image to an external device, and ends an image outputprocess.

As described above, the image generating apparatus 30 of the embodiment1 can generate the reconstructed image in which the composition of anobject has been changed. Moreover, in modifying the composition, thesub-pixels corresponding to the object to be modified are extracted andlinearly-transformed and the sub-pixels not corresponding to the objectto be modified are not transformed. Therefore, the ratio that the pixelof an object using photo shooting information appears in the modifiedconverted and reconstructed image (RI2) is small.

Moreover, in performing the coordinate conversion of pixels to beconverted, when there is a pixel having a large depth coefficient(closer to the main lens ML) at the modification destination, the pixelis prioritized and a converted and reconstructed image is generated.Therefore, according to the depth, the converted and reconstructed imagewhich is close to the converted and reconstructed image in which anobject is actually moved, can be generated.

Moreover, the image generating apparatus 30 of the embodiment 1 has aconfiguration which performs image conversion only for some layers amonga plurality of layers defined with regard to the reconstructed image.This equals to modifying the correspondence relationship only for somepixels whose depth coefficients belong to the range at which an objectto be modified is estimated to be positioned. With this configuration,the image projected on the layer which does not include the object to bemodified is held as it is, and the composition can be modified. For thisreason, an image of an object which is imaged on the light field imageLFI and which is projected on the more distant layer from the main lensML appears in the part before modification. Therefore, a converted andreconstructed image, which is close to the converted and reconstructedimage in which an object is actually moved, can be generated.

Furthermore, as a plurality of layers are defined and a conversionprocess is performed per layer, the process which extracts continuousparts for each layer as the parts to be converted can be performedeasily. Moreover, the difference of “pixels to be converted” and “not tobe converted” can be distinguished clearly and easily.

Furthermore, the digital camera 1 of the embodiment 1 has aconfiguration which tentatively generates a tentative reconstructedimage whose composition is not modified, and presents it to the user.Therefore, the user can choose a desired conversion process, afterchecking the image whose composition is not modified. Therefore, theuser's convenience is highly considered for performing the compositionmodification.

Moreover, the conversion matrix determiner 3250 generates the conversionmatrix for the linear transformation according to the conversionoperation, and the converter 3260 performs image conversion using thisconversion matrix. For this reason, the composition modificationdesignated by the user can be achieved by a small amount of calculation.

Furthermore, the corresponding information based on a photo shootingparameter (shooting setting information) is acquired, and a tentativereconstructed image and a converted and reconstructed image aregenerated based on this corresponding information. Therefore, the highlyprecise tentative reconstructed image and converted and reconstructedimage reflecting the conditions at the time of photo shooting can begenerated.

(Variation Example)

The present invention is not limited to the above-mentioned embodiment,but various variations are possible.

For example, in the above-mentioned embodiment, a plurality of layersfor a reconstructed image are defined, and layer images for each layerare generated. Then, the layer images of the object to be converted(pixels to be converted) are selectively linearly-transformed, and aconverted and reconstructed image is generated by superimposing layers.However, the method of generating a converted and reconstructed image inthe present invention is not limited to this example.

For example, a configuration which does not define a plurality of layersfor a reconstructed image is also possible. In this case, thecorresponding pixels of the pixels to be reconstructed on a sheet ofreconstructed image are extracted, and a tentative reconstructed imageis generated from the pixel value. Then, according to the selectionoperation to the user's pixels to be reconstructed, in the correspondingpixels of the selected pixels to be reconstructed, a corresponding pixelwhose depth coefficient satisfies the predetermined conditions isextracted as a sub-pixel to be converted. Predetermined conditions canbe arbitrarily set. The examples of the predetermined conditions are:the corresponding pixel is included within a range which is within apredetermined range from the greatest numerical value among all thedepth coefficients of the corresponding pixels; and the correspondingpixel is included in a user designated range (not necessarily in a rangeclose to the main lens ML), and the like.

Then, the coordinates of the pixel to be reconstructed corresponding tothe extracted sub-pixels (sub-pixels to be converted) arelinearly-transformed to generate a new correspondence relationship.Then, a converted and reconstructed image is generated based on a newcorrespondence relationship. In the sub-pixels presupposed to correspondto the new correspondence relationship, only the pixel included in thepredetermined range may be extracted. The predetermined range is withina range from the pixel closest to the main lens ML. The reconstructedimage pixel value may be determined from the pixel value of theextracted sub-pixels. There is no necessity of generating areconstructed depth map or a layer depth map in this configuration.

As there is no necessity of defining a layer image for each layeraccording to the configuration, the size of required workspace (memory)can be smaller.

Moreover, in the above-mentioned embodiment, “the pixel on the layerclosest to the main lens ML having a pixel whose pixel value is not NULLat the part which is touched and is in the image on the touch panel” isextracted as pixels to be selected. Then, the continuous pixels on thelayer are extracted based on the pixels to be selected, and the pixelsto be converted are extracted. However, the method for extracting pixelsto be selected or pixels to be converted is not limited to this.

For example, all layer pixels or sub-pixels corresponding to depthcoefficients in the predetermined range designated by a user may beextracted as (sub) pixels to be converted. Or, in the layer pixels orsub-pixels corresponding to reconstructed images positioned in the rangeon a tentative reconstructed image designated by the user, all thepixels having the pixel values in the range designated by the user maybe extracted as (sub) pixels to be converted.

In addition, the user may be warned when the user extracts the pixel ofthe deepest layer (or the deepest sub-pixels) whose pixel value of thepixel to be reconstructed is defined in extracting the pixels to beselected or pixels to be converted. With the described configuration,the user can be noted that there is a possibility that a pixel which isnot based on the photo shooting information due to the compositionmodification may be included. Or a configuration is also possible whicheliminates such a pixel from the object to be extracted.

Furthermore, the pixel conversion information and/or the conversioncontent may not be determined based on the user operation. As otherexamples, a configuration is also possible in which the object of theclosest layer is automatically reduced.

Moreover, in modifying a composition on a layer, the affinetransformation is described as an example. However, not limited to thisexample, any known method for linear transformation can be used toconvert. Moreover, not limited to the linear transformation, the effectsof the present invention can be achieved by using any non-linearconversion in which the area in a reconstructed image occupied by theobject to be converted is varied. The fish eye image correction and thelike can be listed as an example of such image correction.

Moreover, although an example in which the center of conversion is setto the average value (weighted center) of the coordinate value of allthe pixels to be converted is described, not limited to the aboveexample, the center of conversion can be set by performing a coordinateconversion for any coordinate systems. For example, the most far aparttwo points may be extracted in the pixels to be converted, and convertedinto the coordinate system in which the middle point of the most farapart two points are set to the origin to set the middle point to be thecenter of conversion. Moreover, coordinate systems may not be converted.

(Embodiment 2)

The embodiment 2 of the present invention is described.

The image generating apparatus 30 (reconstructed image generatingapparatus) according to the embodiment of the present invention isinstalled in a digital camera 1 as shown in FIG. 1. The similar symbolsare placed to the elements similar to the above embodiment 1 and onlycharacteristic parts are described.

The digital camera 1 has the following functions of i)-vi):

i) a function of photo shooting a light field image which includes aplurality of sub images shot an object from a plurality of viewpoints;

ii) a function of acquiring a depth coefficient which shows a depth ofan object;

iii) a function of generating a reconstructed image in which the imageof the object is reconstructed from the light field image;

iv) a function of displaying the generated reconstructed image;

v) a function of accepting an operation instructing an object to bedeleted; and

vi) a function of generating a reconstructed image whose composition hasbeen modified according to the accepted operation.

The image generating apparatus 30 takes charge of, especially, “vi) afunction of generating a reconstructed image whose composition has beenmodified according to the accepted operation”.

The digital camera 1 comprises an imager 10, an information processor 20including the image generating apparatus 30, a storage 40, and aninterface (an I/F part) 50, as shown in FIG. 1. The digital camera 1,with the described configuration, acquires light beam information of anobject from outside, and generates a reconstructed image. The digitalcamera 1 generates a reconstructed image from which the object to bedeleted is deleted so that the object to be deleted does not block theother objects (principal objects) for viewing this object.

In addition, “to delete an object to be deleted” includes, not onlycompletely deleting an object to be deleted, but also includes that thelight beam information coming from an object to be deleted decreases theinfluence on a reconstructed image to the level in which principalobjects are not blocked for appreciation.

Then, the configuration of the image generating apparatus 30 isdescribed with reference to FIG. 14A and FIG. 14B.

The image generating apparatus 30 physically includes an informationprocessor 31, a main storage 32, an external storage 33, an input andoutput device 36, and an internal bus 37 as shown in FIG. 14A. The imagegenerating apparatus 30 copies a program 38 and data stored in theexternal storage 33 to the main storage 32 and performs a process forgenerating the deleted and reconstructed image mentioned later byallowing the information processor 31 to execute the program 38 usingthe main storage 32.

The image generating apparatus 30 functions as an input device 350, areconstructed image generator 360, a deletion operation acquirer 370 andan output device 380 with the above physical configuration as shown inFIG. 14B.

The input device 350 is a part which takes charge of a function whichthe image generating apparatus 30 acquires information from each part ofthe digital camera 1. The input device 350 includes a reconstructionsetting acquirer 3510, an LFDM acquirer 3520, an LFI acquirer 3530, anda corresponding information acquirer 3540.

The LFDM acquirer 3520 acquires the light field depth map LFDM from thedepth estimation and correspondence determination unit 220.

The reconstruction setting acquirer 3510 acquires reconstruction settinginformation from the reconstruction setting storage 420. The informationreconstruction setting information includes information which describesthe specific details of reconstruction process and information necessaryfor the image generation process as described below (informationindicating the reconstruction parameter and a threshold value). In thisembodiment, the reconstruction parameter presupposes to generate thereconstructed image through the following processes.

-   i) Using corresponding information, the reconstruction parameter    generates a tentative reconstructed image arranged on a    reconstruction plane RE-   ii) The reconstruction parameter displays a tentative reconstructed    image and receives an operation specifying a user's object to be    deleted.-   iii) The reconstruction parameter reconstructs an image from which    an object to be deleted has been deleted based on the object to be    deleted specifying operation and outputs.

For this reason, the reconstruction parameter includes information whichspecifies the distance between the focal point of a new image (on areconstruction plane) and the main lens ML (reconstruction distance,distance a1), the setting parameter for extracting pixels correspondingto the object to be deleted, and the like.

The LFI acquirer 3530 acquires the light field image LFI generated bythe image processor 210.

The corresponding information acquirer 3540 acquires correspondinginformation (corresponding list of FIG. 5) from the depth estimation andcorrespondence determination unit 220.

Each part of the input device 350 transmits the acquired information toeach part of the reconstructed image generator 360.

The deletion operation acquirer 370 receives the information which showsthe operation of specifying an object to be deleted from the operator530 of the digital camera 1. The deletion operation acquirer 370transmits the information which shows the received operation specifyingan object to be deleted to the reconstructed image generator 360.

According to the reconstruction setting acquired by the reconstructionsetting acquirer 3510 and the deletion operation information acquired bythe deletion operation acquirer 370, the reconstructed image generator360 generates the deleted and reconstructed image from the light fielddepth map LFDM acquired by the LFDM acquirer 3520, the light field imageLFI acquired by the LFI acquirer 3530 acquired and the correspondinginformation acquired by the corresponding information acquirer 3540.

In this embodiment, a tentative reconstructed image which is areconstructed image whose composition has not been modified istentatively generated and output. When a user performs an operationspecifying an object to be deleted using the tentative reconstructedimage, the deleted and reconstructed image, which is the reconstructedimage from which the object to be deleted has been deleted, isgenerated. The reconstructed depth map which defines the depths of eachpixel is also generated.

The outline of the process which the reconstructed image generator 360performs is described with reference to FIG. 15A through FIG. 15E. Thereconstructed image generator 360 reconstructs a tentative reconstructedimage (RI3), to which the deletion process is not applied, from thelight field image LFI and outputs the tentative reconstructed image(RI3) to the display 520. In the example of FIG. 15A, the tentativereconstructed image (RI3) is an image in which the background (white), a(gray) child in the vicinity of the focal plane RF, and a nearbyobstacle (shaded part) appear.

Then, the user who observed the tentative reconstructed image (RI3)displayed by the display 520 desires to delete the nearby obstacle sothat the obstacle does not block the image of the child forappreciation. The user chooses an object to be deleted by the operationsuch as tracing the obstacle on a touch panel (the display 520 and theoperator 530) as shown in FIG. 15B.

Based on the selection operation, the reconstructed image generator 360extracts and highlights the object to be deleted (FIG. 15C). The userchecks whether or not the currently highlighted object matches thedesired object to be deleted. When matched, the user touches a fixbutton (YES of FIG. 15C), and confirms the object for deletion. On theother hand, NO is touched when the object highlighted for deletion isdifferent from the desired object. In this case, the selection is reset,and the user performs the selection again.

When the object for deletion is fixed, the reconstructed image generator360 generates and displays an image (a deleted and reconstructed image,RI4) in which the object to be deleted has been deleted. FIG. 15D showsan example of the deleted and reconstructed image in which the object tobe deleted has been deleted completely. On the other hand, FIG. 15Eshows an example of the reconstructed image (RI5) in the case in whichthe influence of the information from an object to be deleted isdecreased and only a certain ratio of the information is deleted.

In order to perform such a process, the reconstructed image generator360 includes a tentative reconstructed image generator 3610, a pixelselection extractor 3620, a depth coefficient to be deleted determiner3630, a deletion pixel extractor 3640, an eliminator 3650, and a deletedand reconstructed image generator 3660.

The tentative reconstructed image generator 3610 defines a reconstructedimage on the reconstruction plane RE. Then, the pixel value of the pixelon a reconstructed image (pixel to be reconstructed) and its depthcoefficient are determined as follows, and a tentative reconstructedimage (RI3) and its reconstructed depth map (RDM3) are generated.

-   (1) One of the pixels to be reconstructed is set to a marked pixel,    and corresponding sub-pixels and their weights which correspond to    the coordinates of the marked image are acquired with reference to    the corresponding information.-   (2) The pixel values of the corresponding pixels are acquired with    reference to the light field image LFI.-   (3) An acquired pixel value multiplied by a weighting coefficient is    set to be a corrected pixel value.-   (4) For all the extracted corresponding pixels, corrected pixel    values are calculated and the summed value of the pixel values is    set to be a pixel value of a marked pixel. Normalization in which    the summed value of the pixel values are divided by summed value of    weights may be performed at this moment.-   (5) The depth coefficient of the extracted corresponding pixel is    acquired with reference to the light field depth map LFDM.-   (6) The mode of the acquired depth coefficient is set to the depth    coefficient of the correspondence part of the reconstructed depth    map.-   (7) Pixel values and depth coefficients are determined by performing    (1)-(6) for each pixel to be reconstructed.

The tentative reconstructed image generator 3610 transmits the generatedtentative reconstructed image (RI3) and the reconstructed depth map(RDM3) to the pixel selection extractor 3620.

An example of a tentative reconstructed image (RI3) and a generatedreconstructed depth map (RDM3) are shown in FIG. 16A through FIG. 16D.FIG. 16A shows an example of a tentative reconstructed image (RI3). FIG.16B shows a reconstructed depth map (RDM3) of a tentative reconstructedimage (RI3). The parts illustrated in solid lines (a nearby object OB3)on the tentative reconstructed image (RI3) is the (white) part with alarge depth coefficient on the reconstructed depth map (RDM3).

On the other hand, the part indicated by the alternating long and shortdash line (more distant object OB2) on the tentative reconstructed image(RI3) is shown in gray on the reconstructed depth map (RDM3).

The parts of objects (background OB1) shown in dashed lines on thetentative reconstructed image (RI3) have small depth coefficients. Theseparts are shown in a darker color in the reconstructed depth map (RDM3).

In the deleted and reconstructed image (RI4, FIG. 16C) in which theobject OB2 has been deleted, the object OB1′ which is hidden by OB2 onthe tentative reconstructed image (RI3) appears. In the deleted andreconstructed depth map (RDM4, FIG. 16D) corresponding to the deletedand reconstructed image (RI4), the part corresponding to OB2 is shown inblack color showing the background.

The tentative reconstructed image generator 3610 transmits and outputsthe generated tentative reconstructed image (RI3) to the output device380. Moreover, the tentative reconstructed image generator 3610transmits the tentative reconstructed image (RI3) and the reconstructeddepth map (RDM3) to the pixel selection extractor 3620. The outputdevice 380 stores the tentative reconstructed image (RI3) to the imagestorage 430 to display on the display 520.

The display 520 presents the user the tentative reconstructed image(RI3), and receives an operation in which the operator 530 chooses theobject on the tentative reconstructed image (RI3) from the user. Thedeletion operation acquirer 370 acquires the received operation.

The deletion operation acquirer 370 transmits the information whichshows an operation specifying the part on the tentative reconstructedimage (RI3) (coordinates of the specified part) to the pixel selectionextractor 3620.

The pixel selection extractor 3620 extracts the pixel to bereconstructed corresponding to the part specified by the user (pixel tobe selected) from the tentative reconstructed image (RI3). Specifically,the coordinates (x, y) in the tentative reconstructed image (RI3) of thepart specified by the user are acquired. Furthermore, the depth value ofthe pixel to be selected is acquired. Then, the acquired information iskept as selection information.

An example of the selection information acquired by the pixel selectionextractor 3620 is described with reference to FIG. 17A through FIG. 17C.In the example of FIG. 17A, the background (OB1) whose depth coefficient(depth index) corresponds to 1, objects (OB2 a and OB2 b) whose depthindex correspond to 3, and the object OB3 whose depth index correspondsto 6 are imaged in the tentative reconstructed image (RI3).

If the user performs a touch operation such as a black arrow on thetentative reconstructed image (RI3), the pixel selection extractor 3620generates the selection pixel list shown in FIG. 17B. The selectionpixel list associates and records the number of pixels to be selected,the coordinates of the pixels to be selected, and the depth coefficients(depth index) of the pixels to be selected. In addition, pixels to beselected are pixels to be reconstructed corresponding to the partspecified by the user using a touch operation. The coordinates of thepixels to be selected are the coordinates of the part on which a touchoperation is performed by the user on the reconstructed image. The depthcoefficients of the pixels to be selected are depth coefficients of thereconstructed depth map (RDM3) corresponding to the coordinates of thepixels to be selected.

The pixel selection extractor 3620 transmits the generated selectionpixel list to the deletion depth determiner 3630.

The deletion depth determiner 3630 determines the depth index whichsatisfies predetermined conditions among the depth index included in thetransmitted selection pixel list as the depth coefficients correspondingto the objects to be deleted (deletion depth coefficient).

A deletion depth coefficient may be determined by any method in whichthe deletion depth coefficient is set to a depth index which has anumber which exceeds a predetermined number in the selection pixel list.However, the following method is used in this embodiment. Specifically,the number of pixels to be selected included in the selection pixel listis set to L, the number of depth index of a numerical value d is set toLd, and the ratio Qd is calculated using the following formula (5).Then, the deletion depth whose Qd exceeds a predetermined thresholdvalue is set to be the deletion depth coefficient among the depthindexes included in the selection pixel list. This threshold value isdefined by reconstruction setting information.Qd=Ld/L  (5)

In the example of FIG. 17A and FIG. 17B, depth indexes 3 and 6 arechosen for the deletion depth coefficients. Then, the deletion depthdeterminer 3630 deletes a pixel having a depth index different from thedeletion depth coefficient from the selection pixel list.

The deletion depth determiner 3630 transmits to the deletion pixelextractor 3640 the selection pixel list which the deletion process iscompleted and the information which shows the defined deletion depthcoefficient.

The deletion pixel extractor 3640 compares the depth coefficients ofsub-pixels with the deletion depth coefficients and extracts the pixelsto be deleted, which can be determined that the object to be deleted isimaged, from the sub-pixels based on the result of the comparison. Theprocess of extracting the pixels to be deleted is mentioned later.

Then, the deletion pixel extractor 3640 transmits the tentativereconstructed image, in which the pixel to be reconstructedcorresponding to the extracted pixel to be deleted is emphasized, to theoutput device 380. The user can determine whether or not the desiredobject has been chosen using the image at this moment.

In an example of FIG. 17C, based on the operation of FIG. 17A, theobjects (OB2 a and OB3) corresponding to the pixels whose depth indexesare 3 and 6 are highlighted to present to the user as the objects to bedeleted. Although the depth index of OB2 b corresponds to the deletiondepth coefficient, the OB2 b is not set to be the object to be deletedas the OB2 b does not include a part specified by the user.

If the user performs an operation that checks the output image and thatconfirms an object to be deleted, the deletion pixel extractor 3640transmits the information indicating the extracted pixels to be deleted(coordinates of the pixel to be deleted on the light field image LFI) tothe eliminator 3650.

The eliminator 3650 deletes pixels to be deleted from the light fieldimage LFI. Specifically, the pixel value of the pixel of the coordinatestransmitted from the deletion pixel extractor 3640 on the light fieldimage LFI (pixel to be deleted) is set to NULL. A pixel whose pixelvalue is NULL is excluded from the object of calculation in thegeneration of a deleted and reconstructed image which is mentionedlater.

In addition, in the present embodiment, “setting the pixel value of apixel to be deleted to NULL” is equal to “not handling the pixel valuehaving NULL as a corresponding pixel of the pixel to be reconstructed inthe following processes”. That is, “setting the pixel value of a pixelto be deleted to NULL” means to delete the pixel to be deleted from thecorresponding list (or weights of correspondence are set to 0).

An example of a process which deletes a pixel to be deleted from thelight field image LFI is described with reference to FIG. 18A throughFIG. 18D. A tabular object OB1, a cubic object OB2, and a cylindricalobject OB3 are imaged in the example. FIG. 18A shows a part of the lightfield image LFI acquired by the LFI acquirer 3530. FIG. 18B shows thetentative reconstructed image reconstructed by the tentativereconstructed image generator 3610. Images of each object appear insymmetrical with respect to a point in the light field image LFI.

The light field image LFI is constructed with a plurality of sub imagesin which OB1, OB2, and OB3 are each shot from different angles. The subimages are shown in the squares arranged in a reticular pattern.Moreover, the center lines of each sub image are shown using thealternating long and short dash lines.

When deleting the OB2, the deletion depth determiner 3630 extracts thedepth coefficient of OB2 as the deletion depth coefficient. Then, thedeletion pixel extractor 3640 extracts the sub-pixel included in thepredetermined range (pixel of the horizontally lined parts in FIG. 18A),which is within a range from a deletion depth coefficient, among thecorresponding pixels which correspond to the coordinates of OB2 on thereconstructed image. Then, the eliminator 3650 sets the pixel values ofthe extracted sub-pixels to NULL. FIG. 18C shows a part of the lightfield image LFI in which the deleted pixels which are set to NULL areshown in black (the deleted LFI, DLFI).

When an image is reconstructed using the corresponding list from thedeleted LFI, the deleted and reconstructed image (RI4) from which theobject to be deleted (OB2) is deleted can be generated as shown in FIG.18D. Then, OB1 is displayed at the part of the existed OB2. This isbecause, in addition to the pixels to be deleted, OB1 is imaged amongthe corresponding pixels of the pixels to be reconstructed at the partwhere OB2 existed.

The eliminator 3650 transmits the deleted LFI to the deleted andreconstructed image generator 3660. The deleted and reconstructed imagegenerator 3660 generates the deleted and reconstructed image (RI4) byweighting and adding the pixel value of the pixel whose pixel value isnot set to NULL on the deleted LFI among the corresponding sub-pixelsregistered in the corresponding list for each of the pixels to bereconstructed.

The deleted and reconstructed image generator 3660 transmits thegenerated deleted and reconstructed image to the output device 380. Theoutput device 380 stores the transmitted deleted and reconstructed image(RI4) in the image storage 430. Then, the digital camera 1 displays thedeleted and reconstructed image (RI4) on the display 520 or transmits toan external device through the I/O device 510.

Then, the process performed by the digital camera 1 is described withreference to flow charts. The digital camera 1 starts the image outputprocess shown in FIG. 19 taking the opportunity that the user operatesthe operator 530 to instruct imaging an object.

In an image output process, the light field image LFI is first generatedfrom the imaging information acquired by the image processor 210 fromthe imager 10, and the shooting setting information acquired from theimage setting storage 410 (Step S501). Then, the digital camera 1transmits the generated light field image LFI and the generated shootingsetting information to the depth estimation and correspondencedetermination unit 220.

Then, the depth estimation and correspondence determination unit 220acquires shooting setting information. Furthermore, the depth estimationand correspondence determination unit 220 acquires reconstructionsetting information from the reconstruction setting storage 420 (StepS502). The imaging setting and reconstruction setting acquired at themoment are also transmitted to the image generating apparatus 30.

Then, the depth estimation and correspondence determination unit 220estimates depths for each sub-pixel which constructs the light fieldimage LFI. Moreover, the image generating apparatus 30 acquires thelight field depth map LFDM in which the depth coefficients are arranged(Step S503).

Specifically, the depth estimation and correspondence determination unit220 estimates the depth of the corresponding object for each sub-pixelconstructing the light field image LFI, and generates the light fielddepth map LFDM in which the depth coefficients indicating the estimateddepth are associated with the sub-pixels.

The depth coefficient may be calculated using any method which estimatesthe distance of the object of the pixel of the light field image LFI.However, in the present embodiment, the depth coefficient is calculatedin a method similar to the S103 in FIG. 10.

Then, the depth estimation and correspondence determination unit 220generates the corresponding list shown in FIG. 5 by the light beam tracedescribed in FIG. 4 (Step S504). Then, the image generating apparatus 30acquires the corresponding list generated by the depth estimation andcorrespondence determination unit 220.

After acquiring the light field depth map LFDM at Step S503 and thecorresponding list at Step S504, the image generating apparatus 30starts a process which generates a tentative reconstructed image (RI3)using the light field image LFI, the light field depth map LFDM, and thecorresponding list (tentative reconstructed image generation process)(Step S505). Here, a tentative reconstructed image generation process 2shown in FIG. 20 is performed.

In the tentative reconstructed image generation process 2, the tentativereconstructed image generator 3610 defines a tentative reconstructedimage first on the reconstruction plane RF defined by the reconstructionsetting information (Step S511). The tentative reconstructed imagegenerator 3610 further selects a marked pixel from the pixel included inthe tentative reconstructed image (pixel to be reconstructed) (StepS512).

Then, the tentative reconstructed image generator 3610 extracts thecorresponding pixels corresponding to the marked pixel from thesub-pixels (Step S513). Specifically, in the corresponding list shown inFIG. 5, the sub-pixels which are registered in the row in which thecoordinates of the pixel to be reconstructed match with the coordinatesof the marked image are extracted as the corresponding pixels.

Then, the tentative reconstructed image generator 3610 calculates thepixel value of the marked pixel (Step S514). Specifically, an arithmeticweighted mean of the pixel values of each pixel extracted at Step S513is acquired using the weights of each corresponding pixel and theacquired value is set to be the pixel value of the marked pixel.

The tentative reconstructed image generator 3610 calculates the depthcoefficient of the marked pixel in a similar manner (Step S515).Specifically, an arithmetic weighted mean of the depth coefficientsdefined by the light field depth map LFDM is acquired for eachcorresponding pixel extracted at Step S513 using the weights of eachcorresponding pixel and the acquired value is set to be the depthcoefficient of the marked layer pixel.

In addition, the mode of the depth coefficient of the correspondingpixel may also set to be the depth coefficient of the marked pixel.

Then, whether or not the above-mentioned process has been completed isdetermined by setting all the pixels to be reconstructed of the markedlayer to the marked pixels (Step S516). When there is an unprocessedpixel to be reconstructed (Step S516; NO), the process is repeated fromStep S512 for the next unprocessed pixel to be reconstructed. On theother hand, if all the pixels to be reconstructed have been processed(Step S516; YES), the tentative reconstructed image generation process 2completes.

Returning to FIG. 19, when a tentative reconstructed image is generatedat Step S505, the tentatively reconstructed image generated by theoutput device 380 is then output (Step S506), and the output device 380allows the display 520 to display the reconstructed image.

Then, regarding the tentative reconstructed image displayed at StepS506, the image generating apparatus 30 performs a process in which thedeletion operation operated by a user is accepted and the object to bedeleted and the pixels to be deleted are defined based on the operation,(deletion object defining process, deletion object defining process 1 inthis case) (Step S507).

The deletion object defining process 1 performed at Step S507 isdescribed with reference to FIG. 21. In the deletion object definingprocess 1, the deletion operation acquirer 370 determines first whetheror not the operation by a user which chooses an object to be deleted hasbeen detected (Step S601). Specifically, whether or not the deletionoperation acquirer 370 can detect an operation by the user touching thetentative reconstructed image (RI3) on the touch panel which isconstructed with the display 520 and the operator 530.

When the operation cannot be detected (Step S601; NO), Step S601 isrepeated and the system waits for the detection of the operation.

On the other hand, when a selection operation can be detected (StepS601; YES), the pixel selection extractor 3620 extracts the pixels to bereconstructed (pixels to be selected) which correspond to the selectionpart (Step S602). Specifically, the coordinates of the pixels to bereconstructed corresponding to the designated positions are acquiredthrough the touch operation to the touch panel.

Then, the pixel selection extractor 3620 acquires the depth coefficientof pixels to be selected with reference to the tentative reconstructedimage (RI3) and its reconstructed depth map (RDM3) (Step S603). Then,the selection pixel extractor 3620 associates selection pixel numbers,coordinates, and depth coefficients (depth indexes) and stores in theselection pixel list of FIG. 17B.

Then, whether or not the next detected operation is the operation forcompleting selection is determined (Step S604). Specifically, it isdetermined whether or not the operations indicating the completion ofselection, such as lifting a finger from a touch panel and/or performingan operation instructing the completion, are detected. When suchoperations are not detected (Step S604; NO), based on an operation forchoosing the subsequent part, the process is repeated from Step S602.

On the other hand, when an operation completing the selection isdetected (Step S604; YES), the process proceeds to Step S605. At StepS605, a process for defining a deletion depth coefficient (deletiondepth determining process, deletion depth determining process 1 in thiscase) is performed.

The deletion depth determining process 1 is described with reference toFIG. 22. In the deletion depth determining process 1, the deletion depthdeterminer 3630 acquires the depth coefficient (depth index) recorded inthe selection pixel list (Step S701). In the example of FIG. 17B, as 3,6, and 1 are recorded as depth index values, these three depth indexesare acquired.

Then, the deletion depth determiner 3630 chooses one of the depthcoefficients acquired at Step S701 as a marked depth coefficient (StepS702).

Then, the ratio Qd is calculated for the marked depth coefficient usingthe formula (5) (Step S703). For example, 45/200 is calculated as avalue of the ratio Qd for the depth index 3 of FIG. 17B.

The deletion depth determiner 3630 then determines whether or not theratio calculated at Step S703 is equal to or more than a predeterminedthreshold value (Step S704). When the ratio calculated at Step S703 isequal to or more than a predetermined threshold value (Step S704; YES),the deletion depth determiner 3630 sets the marked depth as the deletiondepth coefficient (Step S705). On the other hand, when the ratiocalculated at Step S703 is less than a predetermined threshold value(Step S704; NO), the deletion depth determiner 3630 deletes the markeddepth coefficient from the selection pixel list based on the premisethat the marked depth coefficient is not determined to be a deletiondepth coefficient (Step S706). Specifically, in the selection pixellist, a row in which the depth index matches with the marked depthcoefficient is deleted.

Then, it is determined whether or not the above-mentioned process hasbeen completed for all the depth coefficients acquired at Step S701(Step S707). When there is an unprocessed depth coefficient (Step S707;NO), the process is repeated from Step S702 for the next unprocesseddepth coefficient. On the other hand, if the process is completed forall the depth coefficients (Step S707; YES), the deletion depthdetermining process is completed.

Returning to FIG. 21, when the deletion pixel extractor 3640 defines thedeletion depth coefficients at Step S605, the deletion pixel extractor3640 chooses one of the defined deletion depth coefficients as themarked depth coefficient (Step S606).

Then, the deletion pixel extractor 3640 performs a process in whichpixels to be deleted are extracted from the light field image LFI(deletion pixel extraction process, deletion pixel extraction process 1in this case) (Step S607).

The deletion pixel extraction process 1 performed at Step S607 isdescribed with reference to FIG. 23. First, in the deletion pixelextraction process 1, the deletion pixel extractor 3640 chooses one ofthe pixels to be selected as the marked pixel to be selected. The pixelto be selected is stored in the selection pixel list which is generatedfor the tentative reconstructed image (RI3) (Step S801). Then, themarked pixel to be selected is incorporated in the selection area chosenwithin the tentative reconstructed image (RI3).

Then, the deletion pixel extractor 3640 extracts the adjacent pixels tobe reconstructed which are adjacent to the selection area (Step S802).Specifically, the deletion pixel extractor 3640 extracts the pixels tobe reconstructed which are not included in the selection area asadjacent pixels to be reconstructed for each of the pixels included inthe selection area among the pixels bordering to the left, right, up anddown directions.

Then, the deletion pixel extractor 3640 determines if there is a pixelto be incorporated, which satisfies predetermined conditions, among theadjacent pixels to be reconstructed extracted at Step S802 (Step S803).The predetermined condition at this step is that the differences betweenthe pixel values of the adjacent pixels to be reconstructed and theaverage value of the pixels included in the selection area are less thanthe predetermined threshold value set by the reconstruction setting, andthe depth coefficients are included in the predetermined range (deletiondepth range) which includes the deletion depth coefficient (marked depthcoefficient). For example, when the reconstruction setting sets thedeletion depth range as ±2 of the deletion depth coefficient and a depthcoefficient is set to 6, 4-8 is the deletion depth range. In the presentembodiment, “that the depth coefficient of an adjacent pixel to bereconstructed is included in the deletion depth range” becomes acondition for the adjacent pixel to be reconstructed to be a pixel to beincorporated that is incorporated in the selection area.

When there is a pixel to be incorporated among the adjacent pixels to bereconstructed extracted at Step S802 (Step S803; YES), the deletionpixel extractor 3640 incorporates the pixel to be incorporated into theselection area based on the premise that the pixel to be incorporated isdetermined to be a pixel corresponding to an object to be deleted (StepS804). Then, the process returns to Step S802 and extracts furtherpixels to be incorporated.

On the other hand, when there is no incorporated pixel (Step S803; NO),the process proceeds to Step S805 based on the premise that no suchpixel corresponding to an object to be deleted is determined to exist inthe range bordering the selection area at this time.

At Step S805, the deletion pixel extractor 3640 determines if there is apixel which is not included in the selection area among the pixels to beselected stored in the selection pixel list. When there is a pixel to beselected which is not included in the selection area (Step S805; YES),the process is repeated from S801 setting the pixel as a new markedpixel to be selected.

On the other hand, when there is no pixel to be selected which is notincluded in the selection area (Step S805; NO), the deletion pixelextractor 3640 extracts the corresponding pixel from the light fieldimage LFI which corresponds to the selection area of the tentativereconstructed image (RI3) based on the premise that a pixel to bereconstructed corresponding to the object to be deleted is determined tobe included in the present selection area (Step S806). Specifically, thedeletion pixel extractor 3640 extracts, from the corresponding list, thesub-pixel which is defined to correspond to the pixel to bereconstructed included in the selection area.

Then, the deletion pixel extractor 3640 extracts a pixel having thedepth coefficient included in the deletion depth range as a pixel to bedeleted among the corresponding pixels extracted at Step S806 (StepS807). Then, the deletion pixel extraction process 1 is completed.

Returning to FIG. 21, when a pixel to be deleted is extracted at StepS607, then, it is determined whether or not the above-mentioned processis completed for all the deletion depths defined at Step S605 (StepS608). When there is an unprocessed deletion depth (Step S608; NO), theprocess is repeated from Step S606 for the next unprocessed deletiondepth. On the other hand, if the process is completed for all thedeletion depths (Step S608; YES), the process proceeds to Step S609.

At Step S609, as shown in FIG. 15C, the deletion pixel extractor 3640outputs the tentative reconstructed image, in which the object to bedeleted corresponding to the extracted pixels to be deleted ishighlighted, to the output device 380. Specifically, the deletion pixelextractor 3640 generates an image in which the object to be deleted canbe distinguished through processes such as setting the pixel value ofthe object to be deleted to black or setting the pixel value to blinkingblack and white in the tentative reconstructed image (RI3). Then, thegenerated image is displayed.

Then, the user determines whether or not the object to be deleted(pixels to be deleted) should be confirmed by checking the highlighteddeletion area (object to be deleted) and performing a fixing operationand the like (Step S610). When the object to be deleted is not fixed asin the case in which the user chooses NO in FIG. 15C (Step S610; NO),the deletion area is reset and the process returns to Step S601.

On the other hand, when the object to be deleted is fixed as in the casein which the user chooses YES in FIG. 15C (step S610; YES), the deletionpixel extractor 3640 fixes the sub-pixels extracted at Step S607 aspixels to be deleted. Then, the deletion object defining process 1 iscompleted.

Returning to FIG. 19, when a pixel to be deleted is extracted at StepS507, then, the eliminator 3650 performs the deletion process.Specifically, the pixel values of the pixels to be deleted in the lightfield image LFI are set to NULL (Step S508). Then, the deleted andreconstructed image generator 3660 starts the process which generatesthe deleted and reconstructed image (RI4) (deleted and reconstructedimage generation process, deleted and reconstructed image generationprocess 1 in this case) (Step S509).

The deleted and reconstructed image generation process 1 performed atStep S509 is described with reference to FIG. 24. In the deleted andreconstructed image generation process 1, the deleted and reconstructedimage generator 3660 chooses one of the pixels to be reconstructed asthe marked pixel (Step S901).

Then, the deleted and reconstructed image generator 3660 extracts thecorresponding pixel corresponding to the marked pixel from thesub-pixels (Step S902). Specifically, in the corresponding lists of FIG.5, the deleted and reconstructed image generator 3660 extracts as thecorresponding pixels the sub-pixels that are registered in the row inwhich the coordinates of the pixels to be reconstructed are equal to thecoordinates of the marked image.

Then, the deleted and reconstructed image generator 3660 determineswhether or not the corresponding pixel extracted at Step S902 includes apixel whose pixel value is NULL (pixel to be deleted) (Step S903). Whena pixel to be deleted is included (Step S903; YES), the pixel to bedeleted is excluded from being included in subsequent calculationprocesses (Step S904). On the other hand, when a pixel to be deleted isnot included (Step S903; NO), Step S904 is skipped.

Then, the deleted and reconstructed image generator 3660 calculates thepixel value of the marked pixel (Step S905). Specifically, thearithmetic weighted mean, of the pixel values of each correspondingpixel extracted at Step S902, is acquired by using the weights of eachcorresponding pixel. The acquired value is set to be the pixel value ofthe marked pixel. The pixel to be deleted which is excluded at Step S904is not processed in the calculation process.

When all the corresponding pixels are pixels to be deleted (or the ratiooccupied by the pixels to be deleted is larger than a predeterminedratio), a predetermined default value (for example, black) is set to bethe pixel value of the pixel to be reconstructed. At this moment, theuser is warned that information which was not created during photoshooting appears in the deleted and reconstructed pixel through thedeletion operation.

In a similar manner, the deleted and reconstructed image generator 3660calculates the depth coefficient of the marked pixel (Step S906).Specifically, the arithmetic weighted mean, of the depth coefficientsdefined by the light field depth map LFDM for each corresponding pixelextracted at Step S902, is acquired by using the weights of eachcorresponding pixel. The acquired value is set to be the depthcoefficient of the marked pixel. In addition, the mode of the depthcoefficients of corresponding pixels may be set to the depth coefficientof the marked pixel. The pixel to be deleted is excluded from thecalculation process.

Then, the deleted and reconstructed image generator 3660 determineswhether or not the above-mentioned process is completed by setting allthe pixels to be reconstructed to the marked pixels (Step S907). Whenthere is an unprocessed pixel to be reconstructed (Step S907; NO), theprocess is repeated from Step S901 for the next unprocessed pixel to bereconstructed. On the other hand, if all the pixels to be reconstructedhave been processed (Step S907; YES), the deleted and reconstructedimage generation process 1 is completed as the deleted and reconstructedimage is generated.

Returning to FIG. 19, when the deleted and reconstructed image isgenerated at Step S509, the deleted and reconstructed image generated bythe output device 380 then is output to the image storage 430 (StepS510). Then, the display 520 or the I/O device 510 outputs the convertedand reconstructed image to an external device, and the image outputprocess is completed.

As described above, the image generating apparatus 30 of the embodiment2 can generate the reconstructed image in which the object to be deletedis deleted. Moreover, in order to delete the object, the imagegenerating apparatus 30 extracts the sub-pixels corresponding to theobject to be deleted and determines the pixel value of the reconstructedimage based on the pixel values of other sub-pixels. Therefore, theappearance percentage of the pixel being independent from photo shootinginformation of the principal objects (pixels of the object to bedeleted) is low in the reconstructed image (RI4) after deletion. Thatis, the deleted and reconstructed image can be generated as an imagewhich further reflects the photo shooting information of the principalobjects.

Moreover, the image generating apparatus 30 of the embodiment 2 performsthe deletion process for a pixel having the depth coefficient which canbe determined that the object to be deleted is estimated to be located,and deletes an image of the object to be deleted from the reconstructedimage. With this configuration, the image generating apparatus 30 cankeep the image whose depth does not include the object to be deleted asit is, and the object. For this reason, an image of a more distantobject shot within the light field image LFI appears in the deletedpart. Therefore, the deleted and reconstructed image which is close tothe case in which the object is actually removed can be generated.

Moreover, the image generating apparatus 30 of the embodiment 2generates a reconstructed depth map from the light field depth map LFDM,and determines the depth coefficient of an object based on the depthcoefficient of the pixel to be selected (a part of the pixels to bereconstructed). For this reason, the image generating apparatus 30 canextract the sub-pixels, as pixels to be deleted, which are suitable forthe depth of the determined object to be deleted, and can generate thedeleted and reconstructed image which is close to the case in which theobject is actually eliminated.

Furthermore, the digital camera 1 of the embodiment 2 has aconfiguration in which once the tentative reconstructed image which isnot through the deletion process, can be generated and presented to theuser. Therefore, the user can confirm the image which is not through thedeletion process, and choose the desired object to be deleted.Therefore, it is convenient for the user to perform the deletion of theobject.

Furthermore, the digital camera 1 of the embodiment 2 acquirescorresponding information based on the photo shooting parameter(shooting setting information), and generates the tentativereconstructed image and the deleted and reconstructed image based on thecorresponding information. Therefore, a highly accurate tentativereconstructed image and a deleted and reconstructed image reflecting theconditions at the time of photo shooting can be generated.

(An Example of Modification)

The present invention is not limited to this embodiment, but variousmodifications are possible.

For example, in the present embodiment, a range of a deletion depth isset to a predetermined range in which the depth index extracted by thedeletion operation includes more depth coefficients than thepredetermined ratio. Then, the sub-pixel whose depth coefficient isincluded in the deletion depth range among the sub-pixels correspondingto the object to be deleted, is set to be the pixel to be deleted.

However, the method for extracting the pixels to be deleted of thepresent invention is not limited to this.

For example, the digital camera 1 may set all the sub-pixels belongingto the range of the depth specified by the user as pixels to be deleted.With the configuration, for example, when there is a belt-like obstaclein front of the principal objects, a process in which an obstructiveobstacle is deleted uniformly can be achieved easily.

Or a configuration, which sets “the sub-pixels corresponding to any partspecified by the user and belonging to the depth range specified by theuser” to the pixels to be deleted, can also be achieved.

Moreover, although the deletion pixel extraction process 1 of FIG. 23defines the selection area (area in the reconstructed imagecorresponding to the object to be deleted) in the present embodiment,methods of defining the selection area are not limited to this method.

For example, it is possible to make a configuration in which thereconstructed image is divided into image areas in advance based on thepixel value and the depth coefficient, and areas in the image areas, inwhich the number of the pixels to be selected that are included in theimage areas is more than a predetermined ratio, are defined as theselection area. Any known methods such as the division and integrationmethod, and the graph cutting method, can be used for the method fordividing the image areas.

In addition, the method of dividing the reconstructed image into theimage areas in advance in this way is applicable also to the processwhich defines the deletion depth range. For example, areas, in which thenumber of the pixels to be selected that are included in the image areasis more than a predetermined percentage, are extracted as the selectionarea based on the selection operation. Then, the configuration ispossible which sets the mode of the depth coefficient of the selectionarea to the deletion depth coefficient, and sets the predetermined rangeto the deletion depth range from the depth coefficient.

Moreover, another configuration is also possible in designing a digitalcamera which allows the division of a range of depth coefficients into aplurality of classes (depth classes), and performs a deletion process.In the example of the modification, the depth classes are defined usinga list which associates and stores the depth classes and thecorresponding ranges of the depth coefficients (the class-depthcorresponding list of FIG. 25). In the example of FIG. 25, three depthclasses are defined for the depth indexes (0-MAX) assumed in the design.For example, that the class index 2 and the correspondence depth indexes3-5 are registered in the same row indicates that the depth index (depthcoefficient) whose pixels are 3-5 belongs to the second class.

The methods for sorting depths into classes are optional such as amethod, defined in advance with the reconstruction setting, in whicheach depth value of the light field depth map LFDM is clustered by theknown clustering method and the like. Examples of clustering methods maybe a clustering method using the Ward method so that the distancebetween the classes of the depth value to which each class belongs mayserve as the maximum, a clustering method using the k average method andthe like.

With the configuration using a class, the selection pixel list as shownin FIG. 26 is generated based on the selection operation as shown inFIG. 17A. Then, the class index which exceeds a predetermined ratioamong all the pixels to be selected is set to be a selected class. Then,a pixel to be deleted is extracted by setting the depth range allocatedto the selected class as the deletion depth range.

In this configuration, the ratio for selection is calculated for theallocated range as one class. Therefore, even when the object to bedeleted's depth index does not have a fixed numerical value andfluctuates, the user can choose and delete the desired object to bedeleted.

Furthermore, in the present embodiment, the eliminator sets the pixelvalue of the pixel to be deleted to NULL, and excludes the pixel valuefrom the calculation process in generating the deleted and reconstructedimage.

However, in the present invention, the deleted and reconstructed imagemay be generated using any method which reduces the percentage ofappearance of the pixel value of the pixel to be deleted in the pixel tobe reconstructed to an amount suitable for viewing the desired object.

For example, the deleted and reconstructed image can be generated by aconfiguration in which the eliminator eliminates the pixel to be deletedfrom the corresponding list of FIG. 5 such that the pixel value of thepixel to be deleted cannot be used in the generation process of thereconstructed image. Or another configuration is also possible whichreduces the weight “w” stored in the corresponding list for the pixel tobe deleted to a degree suitable for the appreciation of the desiredobject instead of deleting the pixel to be deleted from thecorresponding list. In this configuration, the image in which the pixelsto be deleted are deleted by a certain ratio can be generated as shownin FIG. 15E.

As an example of the configuration, it is possible that the userspecifies the ratio which decreases the weights of the pixels to bedeleted by using a slide bar and the like. With this configuration, theuser can reduce the degree of appearance of the object to be deleted tothe desired level.

Thus, the eliminator performs processes such as setting the pixel valueof a pixel to be deleted to NULL and eliminating them from the object tobe processed in the subsequent reconstruction process, deleting thepixel to be deleted from the corresponding list, and decreasing theweight of the pixel to be deleted and the like. These processes can bereworded as changing the correspondence relationship between the pixelsto be reconstructed and the sub-pixels so that the degree of thecorrespondence between the pixels to be deleted and the pixels to bereconstructed is reduced. In addition, “the degree of correspondence isreduced” includes setting the degree of correspondence to 0. For thisreason, the eliminator can be expressed also as a modifier.

Through these processes, the image generating apparatus can effectivelydecrease the percentage of appearance of the pixel values of thesub-pixels, in which the object to be deleted is determined to beimaged, in the reconstructed image. As a result, as the image ofcorresponding pixels other than the pixels to be deleted appears at apart where the object to be deleted exists, the deleted andreconstructed image close to the image from which the object to bedeleted is actually removed can be generated.

Furthermore, the object to be deleted may not be determined based on theuser operation. For example, another configuration is also possiblewhich automatically deletes the object of the closest class and/or theobject in front of a desired focal plane.

In the embodiment 1 and the embodiment 2, the example, in which theweighting coefficient is included in corresponding information, isdescribed. The corresponding information may be information whichdetermines the sub-pixels corresponding to the pixel to be reconstructedand the weighting coefficient needs not be defined.

Moreover, in the above-mentioned description, the correspondinginformation is generated for each light field image LFI based on theshooting setting information and the reconstruction setting. However,not being limited to the above example, predetermined correspondinginformation stored in advance may be used. This configuration issuitable for decreasing the amount of calculation while maintaining theaccuracy within the allowable tolerances when the assumed photo shootingparameter has little room to fluctuate and the reconstruction distanceis constant.

Moreover, in the above-mentioned description, although the image isdescribed as a gray scale image, the image to be processed in thepresent invention is not limited to a gray scale image. For example, theimage may be an RGB image in which three pixel values, R (red), G(green), and B (blue) are defined for each pixel. In this case, thepixel values are similarly processed as vector values of RGB. Moreover,the above-mentioned process may be performed respectively by settingeach value of R, G, and B as each independent gray scale image. Withthis configuration, the reconstructed image which is a color image canbe generated from the light field image which is a color image.

In addition, the above-mentioned hardware configuration and the flowcharts are an example, and can optionally be altered and modified.

The central portion including the information processor 31, the mainstorage 32, the external storage 33 and the like, which performs aprocess for generating the reconstructed image can be realized by usingtypical computer systems not relying on a dedicated system. For example,the computer program for executing the above-mentioned operation may bestored and distributed in computer readable recording media (a flexibledisk, CD-ROM, DVD-ROM, and the like.), the above-mentioned computerprogram may be installed in a computer, and the central portion, whichperforms a process for generating the reconstructed image, may beconstructed. Moreover, the above-mentioned computer program may bestored in the memory storage of the server apparatuses on acommunication network such as the Internet, and an image generatingapparatus may be configured to be downloaded by usual computer systemsand the like.

An application program portion only may be stored in a recording mediumor a storage when the functions of the image generating apparatus arerealized by the sharing of an operating system(OS) and an applicationprogram, or collaboration of the OS and the application program.

Moreover, it is also possible to superimpose a computer program on acarrier wave and to distribute through a communication network. Forexample, the above-mentioned computer program may be posted on thebulletin board (Bulletin Board System) on a communication network, andthe above-mentioned computer program may be distributed through anetwork. Then, the computer program may be configured such that theabove-mentioned process can be performed by starting the computerprogram and performing other application programs under the control ofthe OS in a similar manner.

As mentioned above, although the preferred embodiment of the presentinvention is described, the present invention is not limited to thespecific embodiment. The present invention includes the inventionsdescribed in the claims and their equivalent scopes.

Having described and illustrated the principles of this application byreference to one or more preferred embodiments, it should be apparentthat the preferred embodiments may be modified in arrangement and detailwithout departing from the principles disclosed herein and that it isintended that the application be construed as including all suchmodifications and variations insofar as they come within the spirit andscope of the subject matter disclosed herein.

What is claimed is:
 1. An image generating apparatus comprising: animage acquirer which acquires a photo shooting image formed by aplurality of sub images shot from each of a plurality of viewpoints; anextractor which extracts, from sub-pixels forming the sub images,sub-pixels corresponding to an object whose composition is to bemodified as pixels to be modified; a modifier which modifies acorrespondence relationship with sub-pixels extracted as the pixels tobe modified among correspondence relationships between the sub-pixelsand pixels to be reconstructed forming a reconstructed image which isdefined on a predetermined reconstruction plane; a generator whichgenerates a reconstructed image in which the composition of the objecthas been modified by calculating pixel values of the pixels to bereconstructed from pixel values of the sub-pixels using the modifiedcorrespondence relationship; and a depth coefficient acquirer whichacquires, for each of the sub-pixels, a depth coefficient indicating adepth of an object corresponding to the sub-pixel; wherein thegenerator, referring to a correspondence relationship modified by themodifier, compares depth coefficients of the pixels to be modified anddepth coefficients of the sub-pixels which are not the pixels to bemodified, and acquires, giving priority to, the pixel values having lessdeep depth coefficients in acquiring pixel values of pixels to bereconstructed corresponding to the pixels to be modified.
 2. An imagegenerating apparatus comprising: an image acquirer which acquires aphoto shooting image formed by a plurality of sub images shot from eachof a plurality of viewpoints; an extractor which extracts, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectwhose composition is to be modified as pixels to be modified; a modifierwhich modifies a correspondence relationship with sub-pixels extractedas the pixels to be modified among correspondence relationships betweenthe sub-pixels and pixels to be reconstructed forming a reconstructedimage which is defined on a predetermined reconstruction plane; agenerator which generates a reconstructed image in which the compositionof the object has been modified by calculating pixel values of thepixels to be reconstructed from pixel values of the sub-pixels using thecorrespondence relationship modified by the modifier; and a depthcoefficient acquirer which acquires, for each of the sub-pixels, a depthcoefficient indicating a depth of an object corresponding to thesub-pixel; wherein the extractor extracts, from the sub-pixelscorresponding to a position of an object to be modified which is anobject whose composition is to be modified, the sub-pixels satisfying apredetermined condition of a depth coefficient obtained by the depthcoefficient acquirer as pixels to be modified corresponding to an objectwhose composition has to be modified.
 3. The image generating apparatusaccording to claim 2, wherein the extractor acquires the depth of theobject to be modified, and sets sub-pixels to satisfy a predeterminedcondition of the depth coefficients, the sub-pixels being included in apredetermined range in which the depth coefficients acquired by thedepth coefficient acquirer include the depth of the object to bemodified.
 4. The image generating apparatus according to claim 3,further comprising a definer which defines, with regards to thereconstructed image defined on the reconstruction plane, a plurality oflayers, each of which are allotted to predetermined ranges of depthcoefficients; wherein the extractor selects a layer corresponding to theobject to be modified among the plurality of layers and sets a rangeallotted to the selected layer to a predetermined range which includesthe depth of the object to be modified.
 5. The image generatingapparatus according to claim 1, further comprising: a second generatorwhich generates a reconstructed image without any compositionmodification as a tentative reconstructed image based on acorrespondence relationship prior to a change by the modifier; an outputdevice which outputs the tentative reconstructed image generated by thesecond generator; and a designator for designating a portion on thetentative reconstructed image output by the output device; wherein theextractor sets an object corresponding to a portion designated by thedesignator to be an object whose composition has to be modified.
 6. Theimage generating apparatus according to claim 1, further comprising: athird acquirer which acquires information indicating details of thecomposition modification; and a determiner which determines amodification parameter to be used for modifying a correspondencerelationship between the pixels to be modified and the pixels to bereconstructed by the modifier based on the details of the compositionmodification indicated by the information acquired by the thirdacquirer; wherein the modifier modifies a correspondence relationshipwith the pixels to be modified using the modification parameterdetermined by the determiner.
 7. The image generating apparatusaccording to claim 6, wherein the modification parameter determined bythe determiner is a conversion matrix for a linear transformation. 8.The image generating apparatus according to claim 1, further comprising:a fourth acquirer which acquires information indicating a correspondencerelationship between the sub-pixels and the pixels to be reconstructed,which are acquired based on a photo shooting parameter when the image isshot; wherein the modifier modifies a correspondence relationshipindicating information acquired by the fourth acquirer.
 9. An imagegenerating apparatus comprising: an image acquirer which acquires aphoto shooting image formed by a plurality of sub images shot from eachof a plurality of viewpoints; an extractor which extracts, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectto be deleted as pixels to be deleted; a modifier which modifies acorrespondence relationship between the sub-pixels and pixels to bereconstructed forming a reconstructed image which is defined on apredetermined reconstruction plane so that a degree of correspondencewith the sub-pixels extracted as the pixels to be deleted is reduced;and a generator which generates a reconstructed image having a smalleffect of the pixels to be deleted by calculating pixel values of thepixels to be reconstructed from pixel values of the sub-pixels using thecorrespondence relationship modified by the modifier; wherein thegenerator generates a reconstructed image from which the object to bedeleted is deleted using the correspondence relationship modified by themodifier.
 10. An image generating apparatus comprising: an imageacquirer which acquires a photo shooting image formed by a plurality ofsub images shot from each of a plurality of viewpoints; an extractorwhich extracts, from sub-pixels forming the sub images, sub-pixelscorresponding to an object to be deleted as pixels to be deleted; amodifier which modifies a correspondence relationship between thesub-pixels and pixels to be reconstructed forming a reconstructed imagewhich is defined on a predetermined reconstruction plane so that adegree of correspondence with the sub-pixels extracted as the pixels tobe deleted is reduced; and a generator which generates a reconstructedimage having a small effect of the pixels to be deleted by calculatingpixel values of the pixels to be reconstructed from pixel values of thesub-pixels using the correspondence relationship modified by themodifier; and a depth coefficient acquirer which acquires, for each ofthe sub-pixels, a depth coefficient indicating a depth of an objectcorresponding to the sub-pixel; wherein the extractor extractssub-pixels which satisfy a predetermined condition of a depthcoefficient acquired by the depth coefficient acquirer from thesub-pixels as pixels to be deleted which corresponds to the object to bedeleted.
 11. The image generating apparatus according to claim 10,wherein the extractor acquires the depth of the object to be deleted andsets sub-pixels to satisfy a predetermined condition of the depthcoefficients, the sub-pixels being included in a predetermined range inwhich the depth coefficients acquired by the depth coefficient acquirerinclude the depth of the object to be deleted.
 12. The image generatingapparatus according to claim 11, further comprising an acquirer whichacquires the depths of the pixels to be reconstructed from depthcoefficients of sub-pixels acquired by the depth coefficient acquirer;wherein the extractor acquires at least one part of a portion within thereconstructed image where the object to be deleted is positioned, andacquires the depth of the object to be deleted based on the depthacquired by the depth coefficient acquirer for the pixels to bereconstructed of the acquired portion.
 13. The image generatingapparatus according to claim 9, further comprising: a second generatorwhich generates a tentative reconstructed image based on acorrespondence relationship prior to the modification by the modifier;an output device which outputs the tentative reconstructed imagegenerated by the second generator; and a designator for designating aportion in the tentative reconstructed image output by the outputdevice; wherein the extractor extracts sub-pixels corresponding to theportion designated by the designator as the pixels to be deleted. 14.The image generating apparatus according to claim 9, further comprising:a corresponding information acquirer which acquires informationindicating a correspondence relationship between the sub-pixels and thepixels to be reconstructed, the information being acquired based on aphoto shooting parameter when the image is shot; wherein the modifiermodifies the correspondence relationship indicated by the informationwhich is acquired by the corresponding information acquirer.
 15. Amethod comprising: acquiring a photo shooting image formed by aplurality of sub images shot from each of a plurality of viewpoints;extracting, from sub-pixels forming the sub image, sub-pixelscorresponding to an object whose composition is to be modified as pixelsto be modified; modifying a correspondence relationship with sub-pixelsextracted as the pixels to be modified among correspondencerelationships between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane; generating a reconstructed image in which thecomposition of the object has been modified by calculating pixel valuesof the pixels to be reconstructed from pixel values of the sub-pixelsusing the modified correspondence relationship; and acquiring, for eachof the sub-pixels, a depth coefficient indicating a depth of an objectcorresponding to the sub-pixel; wherein generating the reconstructedimage comprises referring to the modified correspondence relationshipand comparing depth coefficients of the pixels to be modified and depthcoefficients of the sub-pixels which are not the pixels to be modified,and acquiring, giving priority to, the pixel values having less deepdepth coefficients in acquiring pixel values of the pixels to bereconstructed corresponding to the pixels to be modified.
 16. A methodcomprising: acquiring a photo shooting image formed by a plurality ofsub images shot from each of a plurality of viewpoints; extracting, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectto be deleted as pixels to be deleted; modifying a correspondencerelationship between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane so that a degree of correspondence with thesub-pixels extracted as the pixels to be deleted is reduced; andgenerating a reconstructed image having a small effect of the pixels tobe deleted by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship; wherein generating the reconstructed image comprisesgenerating a reconstructed image from which the object to be deleted isdeleted using the modified correspondence relationship.
 17. Anon-transitory computer-readable recording medium having stored thereona program executable by a computer, the program controlling the computerto perform functions comprising: acquiring a photo shooting image formedby a plurality of sub images shot from each of a plurality ofviewpoints; extracting, from sub-pixels forming the sub images,sub-pixels corresponding to an object whose composition is to bemodified as pixels to be modified; modifying a correspondencerelationship with sub-pixels extracted as the pixels to be modifiedamong correspondence relationships between the sub-pixels and pixels tobe reconstructed forming a reconstructed image which is defined on apredetermined reconstruction plane; generating a reconstructed image inwhich the composition of the object has been modified by calculatingpixel values of the pixels to be reconstructed from pixel values of thesub-pixels using the modified correspondence relationship; andacquiring, for each of the sub-pixels, a depth coefficient indicatingthe depth of an object corresponding to the sub-pixel; whereingenerating the reconstructed image comprises referring to the modifiedcorrespondence relationship and comparing depth coefficients of thepixels to be modified and depth coefficients of the sub-pixels which arenot the pixels to be modified, and acquiring, giving priority to, thepixel values having less deep depth coefficients in acquiring pixelvalues of the pixels to be reconstructed corresponding to the pixels tobe modified.
 18. A non-transitory computer-readable recording mediumhaving stored thereon a program executable by a computer, the programcontrolling the computer to perform functions comprising: acquiring aphoto shooting image formed by a plurality of sub images shot from eachof a plurality of viewpoints; extracting, from sub-pixels forming thesub images, sub-pixels corresponding to an object to be deleted aspixels to be deleted; modifying a correspondence relationship betweenthe sub-pixels and pixels to be reconstructed forming a reconstructedimage which is defined on a predetermined reconstruction plane so that adegree of correspondence with the sub-pixels extracted as the pixels tobe deleted is reduced; and generating a reconstructed image having asmall effect of the pixels to be deleted by calculating pixel values ofthe pixels to be reconstructed from pixel values of the sub-pixels usingthe modified correspondence relationship; wherein generating thereconstructed image comprises generating a reconstructed image fromwhich the object to be deleted is deleted using the modifiedcorrespondence relationship.
 19. A method comprising: acquiring a photoshooting image formed by a plurality of sub images shot from each of aplurality of viewpoints; extracting, from sub-pixels forming the subimage, sub-pixels corresponding to an object whose composition is to bemodified as pixels to be modified; modifying a correspondencerelationship with sub-pixels extracted as the pixels to be modifiedamong correspondence relationships between the sub-pixels and pixels tobe reconstructed forming a reconstructed image which is defined on apredetermined reconstruction plane; generating a reconstructed image inwhich the composition of the object has been modified by calculatingpixel values of the pixels to be reconstructed from pixel values of thesub-pixels using the modified correspondence relationship; andacquiring, for each of the sub-pixels, a depth coefficient indicating adepth of an object corresponding to the sub-pixel; wherein extractingthe sub-pixels comprises extracting sub-pixels satisfying apredetermined condition of the depth coefficient.
 20. A methodcomprising: acquiring a photo shooting image formed by a plurality ofsub images shot from each of a plurality of viewpoints; extracting, fromsub-pixels forming the sub images, sub-pixels corresponding to an objectto be deleted as pixels to be deleted; modifying a correspondencerelationship between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane so that a degree of correspondence with thesub-pixels extracted as the pixels to be deleted is reduced; generatinga reconstructed image having a small effect of the pixels to be deletedby calculating pixel values of the pixels to be reconstructed from pixelvalues of the sub-pixels using the modified correspondence relationship;and acquiring, for each of the sub-pixels, a depth coefficientindicating a depth of an object corresponding to the sub-pixel; whereinextracting the sub-pixels comprises extracting sub-pixels satisfying apredetermined condition of the depth coefficient.
 21. A non-transitorycomputer-readable recording medium having stored thereon a programexecutable by a computer, the program controlling the computer toperform functions comprising: acquiring a photo shooting image formed bya plurality of sub images shot from each of a plurality of viewpoints;extracting, from sub-pixels forming the sub images, sub-pixelscorresponding to an object whose composition is to be modified as pixelsto be modified; modifying a correspondence relationship with sub-pixelsextracted as the pixels to be modified among correspondencerelationships between the sub-pixels and pixels to be reconstructedforming a reconstructed image which is defined on a predeterminedreconstruction plane; and a generation function of generating areconstructed image in which the composition of the object has beenmodified by calculating pixel values of the pixels to be reconstructedfrom pixel values of the sub-pixels using the modified correspondencerelationship; and acquiring, for each of the sub-pixels, a depthcoefficient indicating a depth of an object corresponding to thesub-pixel; wherein extracting the sub-pixels comprises extractingsub-pixels satisfying a predetermined condition of the depthcoefficient.
 22. A non-transitory computer-readable recording mediumhaving stored thereon a program executable by a computer, the programcontrolling the computer to perform functions comprising: acquiring aphoto shooting image formed by a plurality of sub images shot from eachof a plurality of viewpoints; extracting, from sub-pixels forming thesub images, sub-pixels corresponding to an object to be deleted aspixels to be deleted; modifying a correspondence relationship betweenthe sub-pixels and pixels to be reconstructed forming a reconstructedimage which is defined on a predetermined reconstruction plane so that adegree of correspondence with the sub-pixels extracted as the pixels tobe deleted is reduced; generating a reconstructed image having a smalleffect of the pixels to be deleted by calculating pixel values of thepixels to be reconstructed from pixel values of the sub-pixels using themodified correspondence relationship, and acquiring, for each of thesub-pixels, a depth coefficient indicating the depth of an objectcorresponding to the sub-pixel; wherein extracting the sub-pixelscomprises extracting sub-pixels satisfying a predetermined condition ofthe depth coefficient.